Methods, systems, and user interface navigation of video content based spatiotemporal pattern recognition

ABSTRACT

A user interface for a media system supports using gestures, such as swiping gestures and taps, to navigate frame-synchronized video clips or video feeds. The detection of the gestures is interpreted as a command to navigate the frame-synchronized content. In one implementation, a tracking system and a trained machine learning system is used to generate the frame synchronized video clips or video feeds. In one implementation, video clips of an event are organized into storylines and the user interface permits navigation between different storylines and within individual storylines.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/229,457, filed Dec. 21, 2018, which claims the benefit ofU.S. provisional patent application Ser. No. 62/646,012, filed Mar. 21,2018. U.S. patent application Ser. No. 16/229,457, in turn, is a bypasscontinuation-in-part of International Application serial numberPCT/US2017/051768, filed Sep. 15, 2017, and published as WO/2018/053257on Mar. 22, 2018, which claims priority to U.S. provisional patentapplication Ser. No. 62/532,744 filed Jul. 14, 2017, U.S.non-provisional patent application Ser. No. 15/586,379 filed May 4,2017, and U.S. provisional patent application Ser. No. 62/395,886 filedSep. 16, 2016. This application is a continuation-in-part of U.S. patentapplication Ser. No. 15/586,379, filed on May 4, 2017 and published asUS/2017/0238055 on Aug. 17, 2017, which claims the benefit of U.S.provisional patent application Ser. No. 62/395,886, filed Sep. 16, 2016.U.S. patent application Ser. No. 15/586,379, filed on May 4, 2017 is acontinuation-in-part of U.S. patent application Ser. No. 14/634,070filed Feb. 27, 2015, which claims the benefit of U.S. provisional patentapplication Ser. No. 62/072,308 filed Oct. 29, 2014 and U.S. provisionalpatent application Ser. No. 61/945,899 filed Feb. 28, 2014. Each of theapplications listed above and elsewhere herein are hereby incorporatedby reference in its entirety as if fully set forth herein.

BACKGROUND Field of the Invention

The present application generally relates to a system and method forperforming analysis of events that appear in live and recorded videofeeds, such as sporting events. In particular, the present applicationrelates to a system and methods for enabling spatiotemporal analysis ofcomponent attributes and elements that make up events within a videofeed, such as of a sporting event, systems for discovering, learning,extracting, and analyzing such events, metrics and analytic resultsrelating to such events, and methods and systems for display,visualization, and interaction with outputs from such methods andsystems.

Description of the Related Art

Live events, such as sports, especially at the college and professionallevels, continue to grow in popularity and revenue as individualcolleges and franchises reap billions in revenue each year. To providevaluable insights and gain a competitive advantage in such endeavors,quantitative methodologies, such as Sabermetrics, have grown inimportance and ubiquity as a valuable augmentation to traditionalscouting methods. However, as no one person can evaluate and accuratelystore all of the information available from the vast volumes of sportinginformation generated on a daily basis, there seldom exists a storehouseof properly coded and stored information reflecting such large volumesof sports information and, even were such information available, thereis lacking the provision of tools capable of mining and analyzing suchinformation.

Systems are now available for capturing and encoding event information,such as sporting event information, such as “X, Y, Z” motion datacaptured by imaging cameras deployed in National Basketball Association(NBA) arenas. However, there are many challenges with such systems,including difficulty handling the data, difficulty transforming X, Y, Zdata into meaningful and existing sports terminology, difficultyidentifying meaningful insights from the data, difficulty visualizingresults, and others. Also, there are opportunities to identify andextract novel insights from the data. Accordingly, a need exists formethods and systems that can take event data captured in video feeds andenable discovery and presentation of relevant events, metrics, analyticresults, and insights.

SUMMARY

In accordance with various exemplary and non-limiting embodiments,methods and systems disclosed herein enable the exploration of eventdata captured from video feeds, the discovery of relevant events (suchas within a video feed of a sporting event), and the presentation ofnovel insights, analytic results, and visual displays that enhancedecision-making, provide improved entertainment, and provide otherbenefits.

In one embodiment, tracking information and a trained machine learningapparatus is used to generate time aligned content channels for a userin which video segments or video feeds have frame-level synchronization.In one embodiment, a user interface supports gesture-based navigation ofthe frame-level synchronized video clip segments or video feeds. Thegestures may include swiping gestures, such as swiping left, swipingright, swiping up, and swiping down. The gestures may also includetapping gestures. The gestures may be used to navigate related content,including examples such as video clips of the same event from differentangles or with augmented content. In some embodiments, media segments(e.g., video clips) for an event are organized into storylines with eachstoryline having storyline elements in which an individual storylineelement corresponds to media display screen to display a media segment.One set of gestures may be used to navigate between storylines andanother set of gestures used to navigate within storylines. In oneembodiment, horizontal swiping is used to navigate between storylines,and tapping and vertical swiping is used to navigate within anindividual storyline.

BRIEF DESCRIPTION OF THE FIGURES

The following detailed description of certain embodiments may beunderstood by reference to the following figures:

FIG. 1 illustrates a technology stack according to an exemplary andnon-limiting embodiment.

FIG. 2 illustrates a stack flow according to an exemplary andnon-limiting embodiment.

FIG. 3 illustrates an exploration loop according to an exemplary andnon-limiting embodiment.

FIG. 4 illustrates a ranking user interface according to an exemplaryand non-limiting embodiment.

FIGS. 5A and 5B illustrate a ranking user interface according to anexemplary and non-limiting embodiment.

FIGS. 6A and 6B illustrate a filters user interface according to anexemplary and non-limiting embodiment.

FIG. 7 illustrates a breakdown user interface according to an exemplaryand non-limiting embodiment.

FIG. 8 illustrates a breakdown user interface according to an exemplaryand non-limiting embodiment.

FIG. 9 illustrates a personalized user interface according to anexemplary and non-limiting embodiment.

FIG. 10 illustrates an alternative video user interface according to anexemplary and non-limiting embodiment.

FIG. 11 illustrates an alternative report according to an exemplary andnon-limiting embodiment.

FIG. 12 illustrates a court comparison view according to an exemplaryand non-limiting embodiment.

FIG. 13 illustrates a court view according to an exemplary andnon-limiting embodiment.

FIG. 14 illustrates a report according to an exemplary and non-limitingembodiment.

FIG. 15 illustrates a detailed depiction of a game according to anexemplary and non-limiting embodiment.

FIG. 16 illustrates querying and aggregation according to an exemplaryand non-limiting embodiment.

FIG. 17 illustrates a hybrid classification process flow according to anexemplary and non-limiting embodiment.

FIG. 18 illustrates test inputs according to an exemplary andnon-limiting embodiment.

FIG. 19 illustrates test inputs according to an exemplary andnon-limiting embodiment.

FIG. 20 illustrates player detection according to an exemplary andnon-limiting embodiment.

FIG. 21 illustrates a process flow according to an exemplary andnon-limiting embodiment.

FIG. 22 illustrates rebounding according to an exemplary andnon-limiting embodiment.

FIG. 23 illustrates scatter rank according to an exemplary andnon-limiting embodiment.

FIGS. 24A and 24B illustrate reports according to an exemplary andnon-limiting embodiment.

FIG. 25 illustrates a quality assurance user interface according to anexemplary and non-limiting embodiment.

FIG. 26 illustrates a quality assurance user interface according to anexemplary and non-limiting embodiment.

FIG. 27 illustrates camera pose detection according to an exemplary andnon-limiting embodiment.

FIG. 28 illustrates camera pose detection according to an exemplary andnon-limiting embodiment.

FIG. 29 illustrates auto-rotoscoping according to an exemplary andnon-limiting embodiment.

FIGS. 30A, 30B, and 30C illustrate scripted storytelling with assetsaccording to an exemplary and non-limiting embodiment.

FIG. 31 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 32 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 33 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 34 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 35 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 36 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 37 illustrates an example according to an exemplary andnon-limiting embodiment.

FIG. 38 illustrates a screen shot according to an exemplary andnon-limiting embodiment.

FIGS. 39A, 39B, 39C, 39D, and 39E illustrate a screen shot according toan exemplary and non-limiting embodiment.

FIG. 39F illustrates a first person process according to an exemplaryand non-limiting embodiment.

FIG. 40 illustrates a screen shot according to an exemplary andnon-limiting embodiment.

FIGS. 41A and 41B illustrate a screen shot according to an exemplary andnon-limiting embodiment.

FIGS. 42A, 42B, and 42C illustrate a screen shot according to anexemplary and non-limiting embodiment.

FIG. 43 illustrates a screen shot according to an exemplary andnon-limiting embodiment.

FIG. 44 illustrates a process flow according to an exemplary andnon-limiting embodiment.

FIG. 45 illustrates systems and methods for parallel processing,synchronization, and failover across multiple streams of a live eventaccording to an exemplary and non-limiting embodiment.

FIG. 46 illustrates systems and methods for player-specific informationin three-dimensional position identification and reconstruction inaccordance with the various embodiments.

FIG. 47 illustrates systems and methods for a machine learning modelincluding active learning and active quality assurance in accordancewith the various embodiments.

FIG. 48 illustrates systems and methods for dynamic incorporation ofdata or content using a software development kit (SDK) according to anexemplary and non-limiting embodiment.

FIG. 49 illustrates systems and methods of machine-automated ormachine-assisted generation of aggregated media clips according to anexemplary and non-limiting embodiment.

FIG. 50 illustrates systems and methods for generation of dynamic videoaccording to an exemplary and non-limiting embodiment.

FIG. 51 illustrates an example client device configured with a videoplayer application according to an exemplary and non-limitingembodiment.

FIG. 52 illustrates an example configuration of a video playerapplication according to an exemplary and non-limiting embodiment.

FIG. 53 illustrates systems and methods for generation of dynamic videoand performing swiping or tapping gestures to navigate content accordingto an exemplary and non-limiting embodiment.

FIG. 54 illustrates an example of a storyline according to an exemplaryand non-limiting embodiment.

FIG. 55 illustrates an example of performing a tapping gesture to skipto elements within a storyline.

FIG. 56 illustrates an example of performing a vertical swiping gesturein a first direction to display additional discovery informationaccording to an exemplary and non-limiting embodiment.

FIG. 57 illustrates an example of performing a vertical swiping gesturein a second direction to toggle modes for media segments according to anexemplary and non-limiting embodiment.

FIG. 58 illustrates an example of performing a horizontal swipinggesture to jump to another storyline of the user interface according toan exemplary and non-limiting embodiment.

DETAILED DESCRIPTION

FIG. 1 illustrates a technology stack 100 indicative of technologylayers configured to execute a set of capabilities, in accordance withan embodiment of the present invention. The technology stack 100 mayinclude a customization layer 102, an interaction layer 104, avisualizations layer 108, an analytics layer 110, a patterns layer 112,an events layer 114, and a data layer 118, without limitations. Thedifferent technology layers or the technology stack 100 may be referredto as an “Eagle” Stack 100, which should be understood to encompass thevarious layers to allow precise monitoring, analytics, and understandingof spatiotemporal data associated with an event, such as a sports eventand the like. For example, the technology stack may provide an analyticplatform that may take spatiotemporal data (e.g., 3D motion capture“XYZ” data) from National Basketball Association (NBA) arenas or othersports arenas and, after cleansing, may perform spatiotemporal patternrecognition to extract certain “events”. The extracted events may be,for example, (among many other possibilities) events that correspond toparticular understandings of events within the overall sporting event,such as “pick and roll” or “blitz.” Such events may correspond to realevents in a game, and may, in turn, be subject to various metrics,analytic tools, and visualizations around the events. Event recognitionmay be based on pattern recognition by machine learning, such asspatiotemporal pattern recognition, and in some cases, may be augmented,confirmed, or aided by human feedback.

The customization layer 102 may allow performing custom analytics andinterpretation using analytics, visualization, and other tools, as wellas optional crowd-sourced feedback for developing team-specificanalytics, models, exports, and related insights. For example, amongmany other possibilities, the customization layer 102 may facilitate ingenerating visualizations for different spatiotemporal movements of afootball player, or group of players and counter movements associatedwith other players or groups of players during a football event.

The interaction layer 104 may facilitate generating real-timeinteractive tasks, visual representations, interfaces, videos clips,images, screens, and other such vehicles for allowing viewing of anevent with enhanced features or allowing interaction of a user with avirtual event derived from an actual real-time event. For example, theinteraction layer 104 may allow a user to access features or metricssuch as a shot matrix, a screens breakdown, possession detection, andmany others using real-time interactive tools that may slice, dice, andanalyze data obtained from the real-time event such as a sports event.

The visualizations layer 108 may allow dynamic visualizations ofpatterns and analytics developed from the data obtained from thereal-time event. The visualizations may be presented in the form of ascatter rank, shot comparisons, a clip view, and many others. Thevisualizations layer 108 may use various types of visualizations andgraphical tools for creating visual depictions. The visuals may includevarious types of interactive charts, graphs, diagrams, comparativeanalytical graphs, and the like. The visualizations layer 108 may belinked with the interaction layer so that the visual depictions may bepresented in an interactive fashion for a user interaction withreal-time events produced on a virtual platform such as the analyticplatform of the present invention.

The analytics layer 110 may involve various analytics and ArtificialIntelligence (AI) tools to perform analysis and interpretation of dataretrieved from the real-time event such as a sports event so that theanalyzed data results in insights that make sense out of the pulled bigdata from the real-time event. The analytics and AI tools may comprisesuch as search and optimization tools, inference rules engines,algorithms, learning algorithms, logic modules, probabilistic tools andmethods, decision analytics tools, machine learning algorithms, semantictools, expert systems, and the like without limitations.

Output from the analytics 110 and patterns layers 112 is exportable bythe user as a database that enables the customer to configure their ownmachines to read and access the events and metrics stored in the system.In accordance with various exemplary and non-limiting embodiments,patterns and metrics are structured and stored in an intuitive way. Ingeneral, the database utilized for storing the events and metric data isdesigned to facilitate easy export and to enable integration with ateam's internal workflow. In one embodiment, there is a unique filecorresponding to each individual game. Within each file, individual datastructures may be configured in accordance with included structuredefinitions for each data type indicative of a type of event for whichdata may be identified and stored. For example, types of events that maybe recorded for a basketball game include, but are not limited to, isos,handoffs, posts, screens, transitions, shots, closeouts, and chances.With reference to, for example, the data type “screens”, table 1 is anexemplary listing of the data structure for storing information relatedto each occurrence of a screen. As illustrated, each data type iscomprised of a plurality of component variable definitions eachcomprised of a data type and a description of the variable.

TABLE 1 screens id INT Internal ID of this screen. possession_id STRINGInternal ID of the possession in which this event took place. frame INTFrame ID, denoting frame number from the start of the current period.Currently, this marks the frame at which the screener and ballhandlerare closest. frame_time INT Time stamp provided in SportVU data for aframe, measured in milliseconds in the current epoch (i.e., from00:00:00 UTC on 1 January 1970). game_code INT Game code provided inSportVU data. period INT Regulation periods 1-4, overtime periods 5 andup. game_clock NUMBER Number of seconds remaining in period, from 720.00to 0.00. location_x NUMBER Location along length of court, from 0 to 94.location_y NUMBER Location along baseline of court, from 0 to 50.screener INT ID of screener, matches SportVU ID. ballhandler INT ID ofthe ballhandler, matches SportVU ID. screener_defender INT ID of thescreener's defender, matches SportVU ID. ballhandler_defender INT ID ofthe ballhandler's defender, matches SportVU ID. oteam INT ID of team onoffense, matches IDs in SportVU data. dteam INT ID of team on defense,matches IDs in SportVU data. rdef STRING String representing theobserved actions of the ballhandler's defender. sdef STRING Stringrepresenting the observed actions of the screener's defender. scr_typeSTRING Classification of the screen into take, reject, or slip.outcomes_bhr ARRAY Actions by the ballhandler, taken from the outcomesdescribed at the end of the document, such as FGX or FGM. outcomes_scrARRAY Actions by the screener, taken from the outcomes described at theend of the document, such as FGX or FGM.

These exported files, one for each game, enable other machines to readthe stored understanding of the game and build further upon thatknowledge. In accordance with various embodiments, the data extractionand/or export is optionally accomplished via a JSON schema.

The patterns layer 112 may provide a technology infrastructure for rapiddiscovery of new patterns arising out of the retrieved data from thereal-time event such as a sports event. The patterns may comprise manydifferent patterns that corresponding to an understanding of the event,such as a defensive pattern (e.g., blitz, switch, over, under, up totouch, contain-trap, zone, man-to-man, or face-up pattern), variousoffensive patterns (e.g., pick-and-roll, pick-and-pop, horns,dribble-drive, off-ball screens, cuts, post-up, and the like), patternsreflecting plays (scoring plays, three-point plays, “red zone” plays,pass plays, running plays, fast break plays, etc.) and various otherpatterns associated with a player in the game or sports, in each casecorresponding to distinct spatiotemporal events.

The events layer 114 may allow creating new events or editing orcorrecting current events. For example, the events layer may allow forthe analyzing of the accuracy of markings or other game definitions andmay comment on whether they meet standards and sports guidelines. Forexample, specific boundary markings in an actual real-time event may notbe compliant with the guidelines and there may exist some errors, whichmay be identified by the events layers through analysis and virtualinteractions possible with the platform of the present invention. Eventsmay correspond to various understandings of a game, including offensiveand defensive plays, matchups among players or groups of players,scoring events, penalty or foul events, and many others.

The data layer 118 facilitates management of the big data retrieved fromthe real-time event such as a sports event. The data layer 118 may allowcreating libraries that may store raw data, catalogs, corrected data,analyzed data, insights, and the like. The data layer 118 may manageonline warehousing in a cloud storage setup or in any other manner invarious embodiments.

FIG. 2 illustrates a process flow diagram 200, in accordance with anembodiment of the present invention. The process 200 may includeretrieving spatiotemporal data associated with a sports or game andstoring in a data library at step 202. The spatiotemporal data mayrelate to a video feed that was captured by a 3D camera, such as onepositioned in a sports arena or other venue, or it may come from anothersource.

The process 200 may further include cleaning of the rough spatiotemporaldata at step 204 through analytical and machine learning tools andutilizing various technology layers as discussed in conjunction withFIG. 1 so as to generate meaningful insights from the cleansed data.

The process 200 may further include recognizing spatiotemporal patternsthrough analysis of the cleansed data at step 208. Spatiotemporalpatterns may comprise a wide range of patterns that are associated withtypes of events. For example, a particular pattern in space, such as theball bouncing off the rim, then falling below it, may contribute towardrecognizing a “rebound” event in basketball. Patterns in space and timemay lead to recognition of single events or multiple events thatcomprise a defined sequence of recognized events (such as in types ofplays that have multiple steps).

The recognized patterns may define a series of events associated withthe sports that may be stored in an event datastore at step 210. Theseevents may be organized according to the recognized spatiotemporalpatterns; for example, a series of events may have been recognized as“pick,” “rebound,” “shot,” or like events in basketball, and they may bestored as such in the event datastore 210. The event datastore 210 maystore a wide range of such events, including individual patternsrecognized by spatiotemporal pattern recognition and aggregatedpatterns, such as when one pattern follows another in an extended,multi-step event (such as in plays where one event occurs and thenanother occurs, such as “pick and roll” or “pick and pop” events inbasketball, football events that involve setting an initial block, thenspringing out for a pass, and many others).

The process 200 may further include querying or aggregation or patterndetection at step 212. The querying of data or aggregation may beperformed with the use of search tools that may be operably andcommunicatively connected with the data library or the events datastorefor analyzing, searching, aggregating the rough data, cleansed, oranalyzed data, or events data or the events patterns.

At step 214, metrics and actionable intelligence may be used fordeveloping insights from the searched or aggregated data throughartificial intelligence and machine learning tools.

At step 218, for example, the metrics and actionable intelligence mayconvert the data into interactive visualization portals or interfacesfor use by a user in an interactive manner.

In embodiments, an interactive visualization portal or interface mayproduce a 3D reconstruction of an event, such as a game. In embodiments,a 3D reconstruction of a game may be produced using a process thatpresents the reconstruction from a point of view, such as a first personpoint of view of a participant in an event, such as a player in a game.

Raw input XYZ data obtained from various data sources is frequentlynoisy, missing, or wrong. XYZ data is sometimes delivered with attachedbasic events already identified in it, such as possession, pass,dribble, and shot events; however, these associations are frequentlyincorrect. This is important because event identification further downthe process (in Spatiotemporal Pattern Recognition) sometimes depends onthe correctness of these basic events. For example, if two players' XYpositions are switched, then “over” vs “under” defense would beincorrectly characterized, since the players' relative positioning isused as a critical feature for the classification. Even player-by-playerdata sources are occasionally incorrect, such as associating identifiedevents with the wrong player.

First, validation algorithms are used to detect all events, includingthe basic events such as possession, pass, dribble, shot, and reboundthat are provided with the XYZ data. Possession/Non-possession modelsmay use a Hidden Markov Model to best fit the data to these states.Shots and rebounds may use the possession model outputs, combinedwith 1) projected destination of the ball, and 2) player by playerinformation (PBP) information. Dribbles may be identified using atrained ML algorithm and also using the output of the possession model.These algorithms may decrease the basic event labeling error rate byapproximately 50% or more.

Second, the system has a library of anomaly detection algorithms toidentify potential problems in the data including, but not limited to,temporal discontinuities (intervals of missing data are flagged),spatial discontinuities (objects traveling is a non-smooth motion,“jumping”) and interpolation detection (data that is too smooth,indicating that post-processing was done by the data supplier tointerpolate between known data points in order to fill in missing data).This problem data is flagged for human review so that events detectedduring these periods are subject to further scrutiny.

Spatiotemporal Pattern Recognition

Spatiotemporal pattern recognition 208 is used to automatically identifyrelationships between physical and temporal patterns and various typesof events. In the example of basketball, one challenge is how to turn x,y, z positions of ten players and one ball at twenty-five frames persecond into usable input for machine learning and pattern recognitionalgorithms. For patterns one is trying to detect (e.g., pick & rolls),the raw inputs may not suffice. The instances within each patterncategory can look very different from each other. One, therefore, maybenefit from a layer of abstraction and generality. Features that relatemultiple actors in time are key components to the input. Examplesinclude, but are not limited to, the motion of player one (P1) towardsplayer two (P2), for at least T seconds, a rate of motion of at least Vm/s for at least T seconds and at the projected point of intersection ofpaths A and B, and a separation distance less than D.

In embodiments, an algorithm for spatiotemporal pattern recognition canuse relative motion of visible features within a feed, duration ofrelative motion of such features, rate of motion of such features withrespect to each other, rate of acceleration of such features withrespect to each other, a projected point of intersection of suchfeatures, the separation distance of such features, and the like toidentify or recognize a pattern with respect to visible features in afeed, which in turn can be used for various other purposes disclosedherein, such as recognition of a semantically relevant event or featurethat relates to the pattern. In embodiments, these factors may be basedon a pre-existing model or understanding of the relevance of suchfeatures, such as where values or thresholds may be applied within thepattern recognition algorithm to aid pattern recognition. Thus,thresholds or values may be applied to rates of motion, durations ofmotion, and the like to assist in pattern recognition. However, in othercases, pattern recognition may occur by adjusting weights or values ofvarious input features within a machine learning system, without apre-existing model or understanding of the significance of particularvalues and without applying thresholds or the like. Thus, thespatiotemporal pattern recognition algorithm may be based on at leastone pattern recognized by adjusting at least one of an input type and aweight within a machine learning system. This recognition may occurindependently of any of a priori model or understanding of thesignificance of particular input types, features, or characteristics. Inembodiments, an input type may be selected from the group consisting ofrelative direction of motion of at least two visible features, durationof relative motion of visible features with respect to each other, rateof motion of at least two visible features with respect to each other,acceleration of motion of at least two visible features with respect toeach other, projected point of intersection of at least two visiblefeatures with respect to each other and separation distance between atleast two visible features with respect to each other, and the like.

In embodiments of the present disclosure, there is provided a library ofsuch features involving multiple actors over space and time. In the pastmachine learning (ML) literature, there has been relatively little needfor such a library of spatiotemporal features, because there were fewdatasets with these characteristics on which learning could have beenconsidered as an option. The library may include relationships betweenactors (e.g., players one through ten in basketball), relationshipsbetween the actors and other objects such as the ball, and relationshipsto other markers, such as designated points and lines on the court orfield, and to projected locations based on predicted motion.

Another key challenge is there has not been a labeled dataset fortraining the ML algorithms. Such a labeled dataset may be used inconnection with various embodiments disclosed herein. For example, therehas previously been no XYZ player-tracking dataset that already hashigher level events, such as pick and roll (P&R) events) labeled at eachtime frame they occur. Labeling such events, for many different types ofevents and sub-types, is a laborious process. Also, the number oftraining examples required to adequately train the classifier may beunknown. One may use a variation of active learning to solve thischallenge. Instead of using a set of labeled data as training input fora classifier trying to distinguish A and B, the machine finds anunlabeled example that is closest to the boundary between As and Bs inthe feature space. The machine then queries a human operator/labeler forthe label for this example. It uses this labeled example to refine itsclassifier and then repeats.

In one exemplary embodiment of active learning, the system alsoincorporates human input in the form of new features. These features areeither completely devised by the human operator (and inputted as codesnippets in the active learning framework), or they are suggested intemplate form by the framework. The templates use the spatiotemporalpattern library to suggest types of features that may be fruitful totest. The operator can choose a pattern, and test a particularinstantiation of it, or request that the machine test a range ofinstantiations of that pattern.

Multi-Loop Iterative Process

Some features are based on outputs of the machine learning processitself. Thus, multiple iterations of training are used to capture thisfeedback and allow the process to converge. For example, a firstiteration of the ML process may suggest that the Bulls tend to ice theP&R. This fact is then fed into the next iteration of ML training as afeature, which biases the algorithm to label Bulls' P&R defense as ices.The process converges after multiple iterations. In practice, twoiterations have typically been sufficient to yield good results.

In accordance with exemplary embodiments, a canonical event datastore210 may contain a definitive list of events that the system knowsoccurred during a game. This includes events extracted from the XYZdata, as well as those specified by third-party sources, such as PBPdata from various vendors. The events in the canonical event datastore210 may have game clock times specified for each event. The datastore210 may be fairly large. To maintain efficient processing, it is sharedand stored in-memory across many machines in the cloud. This is similarin principle to other methods such as Hadoop™; however, it is much moreefficient, because in embodiments involving events, such as sportingevents, where there is some predetermined structure that is likely to bepresent (e.g., the 24-second shot clock, or quarters or halves in abasketball game), it makes key structural assumptions about the data.Because the data is from sports games, for example, in embodiments onemay enforce that no queries will run across multiple quarters/periods.Aggregation steps can occur across quarters/periods, but query resultswill not. This is one instantiation of this assumption. Any other domainin which locality of data can be enforced will also fall into thiscategory.

Such a design allows rapid and complex querying across all of the data,allowing arbitrary filters, rather than relying on either 1)long-running processes, or 2) summary data, or 3) pre-computed resultson pre-determined filters.

In accordance with exemplary and non-limiting embodiments, data isdivided into small enough shards that each worker shard has a lowlatency response time. Each distributed machine may have multipleworkers corresponding to the number of processes the machine can supportconcurrently. Query results never rely on more than one shard, since weenforce that events never cross quarter/period boundaries. Aggregationfunctions all run incrementally rather than in batch process so that asworkers return results, these are incorporated into the final answerimmediately. To handle results such as rankings pages, where many rowsmust be returned, the aggregator uses hashes to keep track of theseparate rows and incrementally updates them.

Referring to FIG. 3, an exploration loop may be enabled by the methodsand systems disclosed herein, where questioning and exploration canoccur, such as using visualizations (e.g., data effects, referred to asDataFX in this disclosure), processing can occur, such as to identifynew events and metrics, and understanding emerges, leading to additionalquestions, processing and understanding.

Referring to FIG. 4, the present disclosure provides an instant playerrankings feature as depicted in the illustrated user interface. A usercan select among various types of available rankings 402, as indicatedin the drop down list 410, such as rankings relating to shooting,rebounding, rebound ratings, isolations (Isos), picks, postups,handoffs, lineups, matchups, possessions (including metrics andactions), transitions, plays and chances. Rankings can be selected in amenu element 404 for players, teams, or other entities. Rankings can beselected for different types of play in the menu element 408, such asfor offense, defense, transition, special situations, and the like. Theranking interface allows a user to quickly query the system to answer aparticular question instead of thumbing through pages of reports. Theuser interface lets a user locate essential factors and evaluate talentof a player to make more informed decisions.

FIGS. 5A and 5B show certain basic, yet quite in-depth, pages in thesystems described herein, referred to in some cases as the “Eaglesystem.” This user interface may allow the user to rank players andteams by a wide variety of metrics. This may include identified actions,metrics derived from these actions, and other continuous metrics.Metrics may relate to different kinds of events, different entities(players and teams), different situations (offense and defense) and anyother patterns identified in the spatiotemporal pattern recognitionsystem. Examples of items on which various entities can be ranked in thecase of basketball include chances, charges, closeouts, drives,frequencies, handoffs, isolations, lineups, matches, picks, plays,possessions, postups, primary defenders, rebounding (main and raw), offball screens, shooting, speed/load and transitions.

The Rankings UI makes it easy for a user to understand relative qualityof one row item versus other row items, along any metric. Each metricmay be displayed in a column, and that row's ranking within thedistribution of values for that metrics may be displayed for the user.Color coding makes it easy for the user to understand relative goodness.

FIGS. 6A and 6B show a set of filters in the UI, which can be used tofilter particular items to obtain greater levels of detail or selectedsets of results. Filters may exist for seasons, games, home teams, awayteams, earliest and latest date, postseason/regular season, wins/losses,offense home/away, offensive team, defensive team, layers on the courtfor offense/defense, players off court for offense/defense, locations,offensive or defensive statistics, score differential, periods, timeremaining, after timeout play start, transition/no transition, andvarious other features. The filters 602 for offense may includeselections for the ballhandler, the ballhandler position, the screener,the screener position, the ballhandler outcome, the screener outcome,the direction, the type of pick, the type of pop/roll, the direction ofthe pop/roll, and presence of the play (e.g., on the wing or in themiddle). Many other examples of filters are possible, as a filter canexist for any type of parameter that is tracked with respect to an eventthat is extracted by the system or that is in the spatiotemporal dataset used to extract events. The present disclosure also allowssituational comparisons. The user interface allows a user to search fora specific player that may fit into the offense. The highly accuratedataset and easy to use interface allow the user to compare similarplayers in similar situations. The user interface may allow the user toexplore player tendencies. The user interface may allow locating shotlocations and also may provide advanced search capabilities.

Filters enable users to subset the data in a large number of ways andimmediately receive metrics calculated on the subset. Using multipleloops for convergence in machine learning enables the system to returnthe newly filtered data and metrics in real-time, whereas existingmethods would require minutes to re-compute the metrics given thefilters, leading to inefficient exploration loops (FIG. 3). Given thatthe data exploration and investigation process often require many loops,these inefficiencies can otherwise add up quickly.

As illustrated with reference to FIGS. 6A and 6B, there are many filtersthat may enable a user to select specific situations of interest toanalyze. These filters may be categorized into logical groups,including, but not limited to, Game, Team, Location, Offense, Defense,and Other. The possible filters may automatically change depending onthe type of event being analyzed, for example, Shooting, Rebounding,Picks, Handoffs, Isolations, Postups, Transitions, Closeouts, Charges,Drives, Lineups, Matchups, Play Types, Possessions.

For all event types, under the Game category, filters may includeSeason, specific Games, Earliest Date, Latest Date, Home Team, AwayTeam, where the game is being played Home/Away, whether the outcome wasWins/Losses, whether the game was a Playoff game, and recency of thegame.

For all event types, under the Team category, filters may includeOffensive Team, Defensive Team, Offensive Players on Court, DefendersPlayers on Court, Offensive Players Off Court, Defenders Off Court.

For all event types, under the Location category, the user may be givena clickable court map that is segmented into logical partitions of thecourt. The user may then select any number of these partitions in orderto filter only events that occurred in those partitions.

For all event types, under the Other category, the filters may includeScore Differential, Play Start Type (Multi-Select: Field Goal ORB, FieldGoal DRB, Free Throw ORB, Free Throw DRB, Jump Ball, Live Ball Turnover,Defensive Out of Bounds, Sideline Out of Bounds), Periods, SecondsRemaining, Chance After Timeout (T/F/ALL), Transition (T/F/ALL).

For Shooting, under the Offense category, the filters may includeShooter, Position, Outcome (Made/Missed/All), Shot Value, Catch andShoot (T/F/ALL), Shot Distance, Simple Shot Type (Multi-Select: Heave,Angle Layup, Driving Layup, Jumper, Post), Complex Shot Type(Multi-Select: Heave, Lob, Tip, Standstill Layup, Cut Layup, DrivingLayup, Floater, Catch and Shoot), Assisted (T/F/ALL), Pass From(Player), Blocked (T/F/ALL), Dunk (T/F/ALL), Bank (T/F/ALL), Goaltending(T/F/ALL), Shot Attempt Type (Multi-select: FGA No Foul, FGM Foul, FGXFoul), Shot SEFG (Value Range), Shot Clock (Range), Previous Event(Multi-Select: Transition, Pick, Isolation, Handoff, Post, None).

For Shooting, under the Defense category, the filters may includeDefender Position (Multi-Select: PG, SG, SF, PF, CTR), Closest Defender,Closest Defender Distance, Blocked By, Shooter Height Advantage.

For Picks, under the Offense category, the filters may includeBallhandler, Ballhandler Position, Screener, Screener Position,Ballhandler Outcome (Pass, Shot, Foul, Turnover), Screener Outcome(Pass, Shot, Foul, Turnover), Direct or Indirect Outcome, Pick Type(Reject, Slip, Pick), Pop/Roll, Direction, Wing/Middle,Middle/Wing/Step-Up.

For Picks, under the Defense category, the filters may includeBallhandler Defender, Ballhandler Defender Position, Screener Defender,Screener Defender Position, Ballhandler Defense Type (Over, Under,Blitz, Switch, Ice), Screener Defense Type (Soft, Show, Ice, Blitz,Switch), Ballhandler Defense (Complex) (Over, Under, Blitz, Switch, Ice,Contain Trap, Weak), Screener Defense (Complex) (Over, Under, Blitz,Switch, Ice, Contain Trap, Weak, Up to Touch).

For Drives, under the Offense category, the filters may includeBallhandler, Ballhandler Position, Ballhandler Outcome, Direct orIndirect, Drive Category (Handoff, Iso, Pick, Closeout, Misc.), DriveEnd (Shot Near Basket, Pullup, Interior Pass, Kickout, Pullout,Turnover, Stoppage, Other), Direction, Blowby (T/F).

For Drives, under the Defense category, the filters may includeBallhandler Defender, Ballhandler Defender Position, Help DefenderPresent (T/F), Help Defenders.

For most other events, under the Offense category, the filters mayinclude Ballhandler, Ballhandler Position, Ballhandler Outcome, Director Indirect.

For most other events, under the Defense category, the filters mayinclude Ballhandler Defender, Ballhandler Defender Position.

For Postups, under the Offense category, the filters may additionallyinclude Area (Left, Right, Middle).

For Postups, under the Defense category, the filters may additionallyinclude Double Team (T/F).

The present disclosure provides detailed analysis capabilities, such asthrough the depicted user interface embodiment of FIG. 7. In an exampledepicted in FIG. 7, the user interface may be used to know if a playershould try and ice the pick and roll or not between two players. Filterscan go from all picks, to picks involving a selected player asballhandler, to picks involving that ballhandler with a certainscreener, to the type of defense played by that screener. By filteringdown to particular matchups (by player combinations and actions taken),the system allows rapid exploration of the different options for coachesand players, and selection of preferred actions that had the bestoutcomes in the past. Among other things, the system may give a detailedbreakdown of a player's opponent and a better idea of what to expectduring a game. The user interface may be used to know and highlightopponent capabilities. A breakdowns UI may make it easy for a user todrill down to a specific situation, all while gaining insight regardingfrequency and efficacy of relevant slices through the data.

The events captured by the present system may be capable of beingmanipulated using the UI. FIG. 8 shows a visualization, where adrop-down feature 802 allows a user to select various parameters relatedto the ballhandler, such as to break down to particular types ofsituations involving that ballhandler. These types of “breakdowns”facilitate improved interactivity with video data, including enhancedvideo data created with the methods and systems disclosed herein. Moststandard visualizations are static images. For large and complexdatasets, especially in cases where the questions to be answered areunknown beforehand, interactivity enables the user to explore the data,ask new questions, get new answers. Visualizations may be color codedgood (e.g., orange) to bad (e.g., blue) based on outcomes in particularsituations for easy understanding without reading the detailed numbers.Elements like the sizes of partitions can be used, such as to denotefrequency. Again, a user can comprehend significance from a glance. Inembodiments, each column represents a variable for partitioning thedataset. It is easy for a user to add, remove, and re-arrange columns byclicking and dragging. This makes it easy to experiment with differentvisualizations. Furthermore, the user can drill into a particularscenario by clicking on the partition of interest, which zooms into thatpartition, and redraws the partitions in the columns to the right sothat they are re-scaled appropriately. This enables the user to view therelative sample sizes of the partitions in columns to the right, evenwhen they are small relative to all possible scenarios represented incolumns further to the left. In embodiments, a video icon takes a userto video clips of the set of plays that correspond to a given partition.Watching the video gives the user ideas for other variables to use forpartitioning.

Various interactive visualizations may be created to allow users tobetter understand insights that arise from the classification andfiltering of events, such as ones that emphasize color coding for easyvisual inspection and detection of anomalies (e.g., a generally goodplayer with lots of orange but is bad/blue in one specific dimension).Conventionally, most standard visualizations are static images. However,for large and complex datasets, especially in cases where the questionsto be answered are unknown beforehand, interactivity enables the user toexplore the data, ask new questions, get new answers. For example, abreakdown view may be color coded good (orange) to bad (blue) for easyunderstanding without reading the numbers. Sizes of partitions maydenote the frequency of events. Again, one can comprehend from a glancethe events that occur most frequently. Each column of a visualizationmay represent a variable for partitioning the dataset. It may be easy toadd, remove, and re-arrange columns by clicking and dragging. This makesit easy to experiment with possible visualizations. In embodiments, avideo icon may take a user to video clips, such as of the set of playsthat correspond to that partition. Watching the video gives the userideas for other variables to use for partitioning.

In embodiments, a ranking view is provided. Upon moussing over each rowof a ranking view, histograms above each column may give the user aclear contextual understanding that row's performance for each columnvariable. The shape of a distribution is often informative. Color-codedbars within each cell may also provide a view of each cell's performancethat is always available, without moussing over. Alternatively, thecells themselves may be color-coded.

The system may provide a personalized video in embodiments of themethods and systems described herein. For example, with little time toscout the opposition, the system can provide a user with relevantinformation to quickly prepare the team. The team may rapidly retrievethe most meaningful plays, cut, and compiled to specific needs ofplayers. The system may provide immediate video cut-ups. In embodiments,the present disclosure provides a video that is synchronized withidentified actions. For example, if spatiotemporal machine learningidentifies a segment of a video as showing a pick and roll involving twoplayers, then that video segment may be tagged, so that when that eventis found (either by browsing or by filtering to that situation), thevideo can be displayed. Because the machine understands the precisemoment that an event occurs in the video, a user-customizable segment ofvideo can be created. For example, the user can retrieve videocorresponding to x seconds before, and y seconds after, each eventoccurrence. Thus, the video may be tagged and associated with events.The present disclosure may provide a video that may allow customizationby numerous filters of the type disclosed above, relating to finding avideo that satisfies various parameters, that displays various events,or combinations thereof. For example, in embodiments, an interactiveinterface provided by the present disclosure allows watching videosclips for specific game situations or actions.

Reports may provide a user with easy access to printable pagessummarizing pre-game information about an opponent, scouting report fora particular player, or a post-game summary. For example, the reportsmay collect actionable useful information in one to two easy-to-digestpages. These pages may be automatically scheduled to be sent to otherstaff members, e.g., post-game reports sent to coaches after each game.Referring to FIG. 11, a report may include statistics for a givenplayer, as well as visual representations, such as of locations 1102where shots were taken, including shots of a particular type (such ascatch and shoot shots).

The UI as illustrated in FIG. 12 provides a court comparison view 1202among several parts of a sports court (and can be provided amongdifferent courts as well). For example, filters 1204 may be used toselect the type of statistic to show for a court. The statistics can befiltered to show results filtered by left side 1208 or right side 1214.Where the statistics indicate an advantage, the advantages can be shown,such as of left side advantages 1210 and right side advantages 1212.

In sports, the field of play is an important domain constant orelements. Many aspects of the game are best represented for comparisonon a field of play. In embodiments, a four court comparison view 1202 isa novel way to compare two players, two teams, or other entities, togain an overview view of each player/team (Leftmost and Rightmostfigures) 1208, 1214 and understand each one's strengths/weaknesses (Leftand Right Center figures 1210, 1212).

The court view UI 1302 as illustrated in FIG. 13 provides a court view1304 of a sport arena 1304, in accordance with an embodiment of thepresent disclosure. Statistics for very specific court locations can bepresented on a portion 1308 of the court view. The UI may provide a viewof custom markings, in accordance with an embodiment of the presentinvention.

Referring to FIG. 14, filters may enable users to subset the data in alarge number of ways, and immediately receive metrics calculated on thesubset. Descriptions of particular events may be captured and madeavailable to users.

Various events may be labeled in a game, as reflected in FIG. 15, whichprovides a detailed view of a timeline 1502 of a game, broken down bypossession 1504, by chances 1508, and by specific events 1510 thatoccurred along the timeline 1502, such as determined by spatiotemporalpattern recognition, by human analysis, or by a combination of the two.Filter categories available by a user interface of the presentdisclosure may include ones based on seasons, games, home teams, awayteams, earliest date, latest date, postseason/regular season,wins/losses, offense home/away, offensive team, defensive team, playerson the court for offense/defense, players off court for offense/defense,location, score differential, periods, time remaining, play type (e.g.,after timeout play) and transition/no transition. Events may includeones based on primitive markings, such as shots, shots with a correctedshot clock, rebounds, passes, possessions, dribbles, and steals, andvarious novel event types, such as SEFG (shot quality), EFG+, playeradjusted SEFG, and various rebounding metrics, such as positioning,opportunity percentage, attack, conversion percentage, rebounding aboveposition (RAP), attack+, conversion+ and RAP+. Offensive markings mayinclude simple shot types (e.g., angled layup, driving layup, heave,post shot, jumper), complex shot types (e.g., post shot, heave, cutlayup, standstill layup, lob, tip, floater, driving layup, catch andshoot stationary, catch and shoot on the move, shake & raise, overscreen, pullup and stepback), and other information relating to shots(e.g., catch and shoot, shot clock, 2/3S, assisted shots, shootingfoul/not shooting foul, made/missed, blocked/not blocked,shooter/defender, position/defender position, defender distance and shotdistance). Other events that may be recognized, such as through thespatiotemporal learning system, may include ones related to picks(ballhandler/screener, ballhandler/screener defender, pop/roll,wing/middle, step-up screens, reject/slip/take, direction(right/left/none), double screen types (e.g., double, horns, L, andhandoffs into pick), and defense types (ice, blitz, switch, show, soft,over, under, weak, contain trap, and up to touch), ones related tohandoffs (e.g., receive/setter, receiver/setter defender, handoffdefense (ice, blitz, switch, show, soft, over, or under),handback/dribble handoff, and wing/step-up/middle), ones related toisolations (e.g., ballhandler/defender and double team), and onesrelated to post-ups (e.g., ballhandler/defender, right/middle/left anddouble teams).

Defensive markings are also available, such as ones relating tocloseouts (e.g., ballhandler/defender), rebounds (e.g., players goingfor rebounds (defense/offense)), pick/handoff defense, post doubleteams, drive blow-bys and help defender on drives), ones relating to offball screens (e.g., screener/cutter and screener/cutter defender), onesrelating to transitions (e.g., when transitions/fast breaks occur,players involved on offense and defense, and putback/no putback), onesrelating to how plays start (e.g., after timeout/not after timeout,sideline out of bounds, baseline out of bounds, field goal offensiverebound/defensive rebound, free throw offensive rebound/defensiverebound and live ball turnovers), and ones relating to drives, such asballhandler/defender, right/left, blowby/no blowby, help defenderpresence, identity of help defender, drive starts (e.g., handoff, pick,isolation or closeout) and drive ends (e.g., shot near basket, interiorpass, kickout, pullup, pullout, stoppage, and turnover). These examplesand many others from basketball and other sports may be defined, basedon any understanding of what constitutes a type of event during a game.Markings may relate to off ball screens (screener/cutter),screener/cutter defender, screen types (down, pro cut, UCLA, wedge, widepin, back, flex, clip, zipper, flare, cross, and pin in).

FIG. 16 shows a system 1602 for querying and aggregation. Inembodiments, data is divided into small enough shards that each workerhas low latency response time. Each distributed machine may havemultiple workers corresponding to the number of processes the machinecan support concurrently. Query results never rely on more than oneshard, since we enforce that events never cross quarter/periodboundaries. Aggregation functions all run incrementally rather than inbatch process, so that as workers return results, these are incorporatedinto the final answer immediately. To handle results such as rankingspages, where many rows must be returned, the aggregator uses hashes tokeep track of the separate rows and incrementally updates them.

FIG. 17 shows a process flow for a hybrid classification process thatuses human labelers together with machine learning algorithms to achievehigh accuracy. This is similar to the flow described above in connectionwith FIG. 2, except with the explicit inclusion of the human-machinevalidation process. By taking advantage of aligned video as describedherein, one may provide an optimized process for human validation ofmachine labeled data. Most of the components are similar to thosedescribed in connection with FIG. 2 and in connection with thedescription of aligned video, such as the XYZ data source 1702, cleaningprocess 1704, spatiotemporal pattern recognition module 1712, eventprocessing system 1714, video source 1708, alignment facility 1710 andvideo snippets facility 1718. Additional components include a validationand quality assurance process 1720 and an event-labeling component 1722.Machine learning algorithms are designed to output a measure ofconfidence. For the most part, this corresponds to the distance from aseparating hyperplane in the feature space. In embodiments, one maydefine a threshold for confidence. If an example is labeled by themachine and has confidence above the threshold, the event goes into thecanonical event datastore 210 and nothing further is done. If an examplehas a confidence score below the threshold, then the system may retrievethe video corresponding to this candidate event, and ask a humanoperator to provide a judgment. The system asks two separate humanoperators for labels. If the given labels agree, the event goes into thecanonical event datastore 210. If they do not, a third person, known asthe supervisor, is contacted for a final opinion. The supervisor'sdecision may be final. The canonical event datastore 210 may containboth human marked and completely automated markings. The system may useboth types of marking to further train the pattern recognitionalgorithms. Event labeling is similar to the canonical event datastore210, except that sometimes one may either 1) develop the initial goldstandard set entirely by hand, potentially with outside experts, or 2)limit the gold standard to events in the canonical event datastore 210that were labeled by hand, since biases may exist in the machine labeleddata.

FIG. 18 shows test video input for use in the methods and systemsdisclosed herein, including views of a basketball court from simulatedcameras, both simulated broadcast camera views 1802, as well aspurpose-mounted camera views 1804.

FIG. 19 shows additional test video input for use in the methods andsystems disclosed herein, including input from broadcast video 1902 andfrom purpose-mounted cameras 1904 in a venue. Referring to FIG. 20,probability maps 2004 may be computed based on likelihood there is aperson standing at each x, y location.

FIG. 21 shows a process flow of an embodiment of the methods and systemsdescribed herein. Initially, in an OCR process 2118, machine visiontechniques are used to automatically locate the “score bug” anddetermine the location of the game clock, score, and quarterinformation. This information is read and recognized by OCR algorithms.Post-processing algorithms using various filtering techniques are usedto resolve issues in the OCR. Kalman filtering/HMMs used to detecterrors and correct them. Probabilistic outputs (which measure degree ofconfidence) assist in this error detection/correction. Next, in arefinement process 2120, sometimes, a score bug is nonexistent or cannotbe detected automatically (e.g., sometimes during PIP or split screens).In these cases, remaining inconsistencies or missing data is resolvedwith the assistance of human input. Human input is designed to be sparseso that labelers do not have to provide input at every frame.Interpolation and other heuristics are used to fill in the gaps.Consistency checking is done to verify game clock. Next, in an alignmentprocess, 2112 the Canonical Datastore 2110 (referred to elsewhere inthis disclosure alternatively as the event datastore) contains adefinitive list of events that the system knows occurred during a game.This includes events extracted from the XYZ data 2102, such as aftercleansing 2104 and spatiotemporal pattern recognition 2108, as well asthose specified by third-party sources such as player-by-player datasets 2106, such as available from various vendors. Differences among thedata sources can be resolved, such as by a resolver process. The eventsin the canonical datastore 2110 may have game clock times specified foreach event. Depending on the type of event, the system knows that theuser will be most likely to be interested in a certain interval of gameplay tape before and after that game clock. The system can thus retrievethe appropriate interval of video for the user to watch.

One challenge pertains to the handling of dead ball situations and othergame clock stoppages. The methods and systems disclosed herein includenumerous novel heuristics to enable computation of the correct videoframe that shows the desired event, which has a specified game clock,and which could be before or after the dead ball since those frames havethe same game clock. The game clock is typically specified only at theone-second level of granularity, except in the final minute of eachquarter.

Another advance is to use machine vision techniques to verify some ofthe events. For example, video of a made shot will typically show thescore being increased, or will show a ball going through a hoop. Eitherkind of automatic observation serves to help the alignment processresult in the correct video frames being shown to the end user.

Next, in a query UI component 2130, the UI enables a user to quickly andintuitively request all video clips associated with a set ofcharacteristics: player, team, play type, ballhandler, ballhandlervelocity, time remaining, quarter, defender, etc. In addition, when auser is watching a video clip, the user can request all events that aresimilar to whatever just occurred in the video. The system uses a seriesof cartoon-like illustration to depict possible patterns that represent“all events that are similar.” This enables the user to choose theintended pattern, and quickly search for other results that match thatpattern.

Next, the methods and systems may enable delivery of enhanced video, orvideo snips 2124, which may include rapid transmission of clips fromstored data in the cloud. The system may store video as chunks (e.g.,one-minute chunks), such as in AWS S3, with each subsequent fileoverlapping with a previous file, such as by 30 seconds. Thus, eachvideo frame may be stored twice. Other instantiations of the system maystore the video as different sized segments, with different amounts ofoverlap, depending on the domain of use. In embodiments, each video fileis thus kept at a small size. The 30-second duration of overlap may beimportant because most basketball possessions (or chances in ourterminology) do not last more than 24 seconds. Thus, each chance can befound fully contained in one video file, and in order to deliver thatchance, the system does not need to merge content from multiple videofiles. Rather, the system simply finds the appropriate file thatcontains the entire chance (which in turn contains the event that is inthe query result), and returns that entire file, which is small. Withthe previously computed alignment index, the system is also able toinform the UI to skip ahead to the appropriate frame of the video filein order to show the user the query result as it occurs in that videofile. This delivery may occur using AWS S3 as the file system, theInternet as transport, and a browser-based interface as the UI. It mayfind other instantiations with other storage, transport, and UIcomponents.

FIG. 22 shows certain metrics that can be extracted using the methodsand systems described herein, relating to rebounding in basketball.These metrics include positioning metrics, attack metrics, andconversion metrics. For positioning, the methods and systems describedherein first address how to value the initial position of the playerswhen the shot is taken. This is a difficult metric to establish. Themethods and systems disclosed herein may give a value to the real estatethat each player owns at the time of the shot. This breaks down into twoquestions: (1) what is the real estate for each player? (2) what is itworth? To address the first question, one may apply the technique ofusing Voronoi (or Dirichlet) tessellations. Voronoi tessellations areoften applied to problems involving spatial allocation. Thesetessellations partition a space into Voronoi cells given a number ofpoints in that space. For any point, it is the intersection of theself-containing half-spaces defined by hyper-planes equidistant fromthat point to all other points. That is, a player's cell is all thepoints on the court that are closer to the player than any other player.If all players were equally capable they should be able to control anyrebound that occurred in this cell. One understands that players are notequally capable however this establishment of real estate is to set abaseline for performance. Over performance or under performance of thisbaseline will be indicative of their ability. To address the secondquestion, one may condition based on where the shot was taken andcalculate a spatial probability distribution of where all rebounds forsimilar shots were obtained. For each shot attempt, one may choose acollection of shots closest to the shot location that provides enoughsamples to construct a distribution. This distribution captures thevalue of the real estate across the court for a given shot. To assigneach player a value for initial positioning, i.e., the value of the realestate at the time of the shot, one may integrate the spatialdistribution over the Voronoi cell for that player. This yields thelikelihood of that player getting the rebound if no one moved when theshot was taken and they controlled their cell. We note that because weuse the distribution of locations of the rebound conditioned on theshot, it is not a matter of controlling more area or even necessarilyarea close to the basket, but the most valuable area for that shot.While the most valuable areas are almost always close to the basket,there are some directional effects.

For an attack or hustle metric, one may look at phases following a shot,such as an initial crash phase. To analyze this, one may look at thetrajectory of the ball and calculate the time that it gets closest tothe center of the rim. At this point, one may reapply the Voronoi-basedanalysis and calculate the rebound percentages of each player, i.e., thevalue of the real the estate that each player has at the time the ballhits the rim. The change in this percentage from the time the shot istaken to the time it hits the rim is the value or likelihood the playerhad added during the phase. Players can add value by crashing theboards, i.e., moving closer to the basket towards places where therebound is likely to go, or by blocking out, i.e., preventing otherplayers by taking valuable real estate that is already established. Auseful, novel metric for the crash phase is generated by subtracting therebound probability at the shot from the rebound probability at the rim.The issue is that the ability to add probability is not independent ofthe probability at the shot. Consider a case of a defensive player whoplays close to the basket. The player is occupying high-value realestate, and once the shot is taken, other players are going to startcoming into this real estate. It is difficult for players with highinitial positioning value to have positive crash deltas. Now consider aplayer out by the three-point line. Their initial value is very low andmoving any significant distance toward the rim will give them a positivecrash delta. Thus, it is not fair to compare these players on the samescale. To address this, one may look at the relationship of the rawcrash deltas (the difference between the probability at rim andprobability at shot) compared to the probability at shot. In order tonormalize for this effect, one may subtract the value of the regressionat the player's initial positioning value from the raw crash delta toform the player's Crash value. Intuitively, the value indicates how muchmore probability is added by this player beyond what a player withsimilar initial positioning would add. One may apply this normalizationmethodology to all the metrics the initial positioning affects the otherdimensions and it can be beneficial to control for it.

A player has an opportunity to rebound the ball if they are the closestplayer to the ball once the ball gets below ten feet (or if they possessthe ball while it is above ten feet). The player with the firstopportunity may not get the rebound so multiple opportunities could becreated after a single field goal miss. One may tally the number offield goal misses for which a player generated an opportunity forthemselves and divided by the number of field goals to create anopportunity percentage metric. This indicates the percentage of fieldgoal misses for which that player ended up being closest to the ball atsome point. The ability for a player to generate opportunities beyondhis initial position is the second dimension of rebounding: Hustle.Again, one may then apply the same normalization process as describedearlier for Crash.

The reason that there are often multiple opportunities for rebounds forevery missed shot is that being closest to the ball does not mean that aplayer will convert it into a rebound. Thus, the third dimension ofrebounding, conversion. The raw conversion metric for players iscalculated simply by dividing the number of rebounds obtained by thenumber of opportunities generated.

Formally, given a shot is described by its 2D coordinates on the court,s_x and s_y, which is followed by a rebound r, also described by itscoordinates on the court of r_x and r_y, one may estimate P(r_y,r_x|s_x, s_y), the probability density of the rebound occurring at eachposition on the court given its shot location.

This may be accomplished by first discretizing the court into, forexample, 156 bins, created by separating the court into 13 equallyspaced columns, and 12 equally spaced rows. Then, given some set S ofshots from a particular bin, the rebounds from S will be distributed inthe bins of the court according to a multinomial distribution. One maythen apply maximum likelihood estimation to determine the probability ofa rebound in each of the bins of the court, given the training set S.This process may be performed for bins that shots may fall in, giving156 distributions for the court.

Using these distributions, one may determine P(r_y, r_x|s_x, s_y).First, the shot is mapped to an appropriate bin. The probabilitydistribution determined in the previous step is then utilized todetermine the probability of the shot being rebounded in every bin ofthe court. One assumes that within a particular bin, the rebound isuniformly likely to occur in any coordinate. Thus, a probability densityof the probability of the rebound falling in the bin is assigned to allpoints in the bin.

Using the probability density P(r_y, r_x|s_x, s_y), one may determinethe probability that each particular player grabs the rebound giventheir location and the position of the other players on the court.

To accomplish this, one may first create a Voronoi diagram of the court,where the set of points is the location (p_x, p_y) for each player onthe court. In such a diagram, each player is given a set of points thatthey control. Formally one may characterize the set of points thatplayer P_k controls in the following manner, where X is all points onthe court, and d denotes the Cartesian distance between 2 points.R _(k) ={x∈X|d(x,P _(k))≤d(x,P _(j)) for all j≠k}

Now there exist the two components for determining the probability thateach player gets the rebound given their location, specifically, theshot's location, and the location of all the other players on the court.One may determine this value by assuming that if a ball is rebounded, itwill always be rebounded by the closest available player. Therefore, byintegrating the probability of a rebound over each location in theplayer's Voronoi cell, we determine their rebound probability:∫_(R) P(r _(x) ,r _(y) |s _(x) ,s _(y))dxdy

The preceding section describes a method for determining the playersrebounding probability, assuming that the players are stationary.However, players often move in order to get into better positions forthe rebound, especially when they begin in poor positions. One mayaccount for these phenomena. Let the player's raw rebound probability bedenoted r_(p) and let d be an indicator variable denoting whether theplayer is on defense.

One may then attempt to estimate the player's probability of getting arebound, which we express in the following manner:P(r|rp,d)

One does this by performing two linear regressions, one for theoffensive side of the ball and one for the defensive. One may attempt toestimate p(r|r_(p), d) in the following manner:P(r|r _(p) ,d=0)=A _(o) *r _(p) +B _(o)P(r|r _(p) ,d=1)=A _(d) *r _(p) +B _(d)

This results in four quantities to estimate. One may do this byperforming an ordinary least squares regression for offensive anddefensive players' overall rebounds in the test set. One may use 1 as atarget variable when the player rebounds the ball, and 0 when he doesnot. This regression is performed for offense to determine A_(o) andB_(o) and for defense to determine A_(d) and B_(d). One can then use thevalues to determine the final probability of each player getting therebound given the shots location and the other players on the court.

Novel shooting metrics can also be created using this system. One isable to determine the probability of a shot being made given variousfeatures of the shot s, denoted as F. Formally each shot can becharacterized by a feature vector of the following form.

[dist (hoop, shooter), dist(shooter, defender₀), |angle(hoop, shooter,defender₀)|,|angle(shooter, hoop, hoop_(other)), I(shot=catchAndShoot),dist(shooter, defender₁)]

Here, the hoop represents the basket the shooter is shooting at,defender₀ refers to the closest defender to the shooter, defender₁refers to the second closest defender, and hoop_(other) refers to thehoop on the other end of the court. The angle function refers to theangle between three points, with the middle point serving as the vertex.I(shot=catchAndShoot) is an indicator variable, set to 1 if the shootertook no dribbles in the individual possession before shooting the shot,otherwise set to 0.

Given these features, one seeks to estimate P(s=make). To do this, onemay first split the shots into 2 categories, one for where dist (hoop,shooter) is less than 10, and the other for the remaining shots. Withineach category one may find coefficients β₀, β₁, . . . , β₅ for thefollowing equation:1/(1+e{circumflex over ( )}(−t))wheret=F ₀*_(β) ₀ +F ₁*_(β) ₁ + . . . +F ₅*_(β) ₅

Here, F₀ through F₅ denote the feature values for the particular shot.One may find the coefficient values β₀, β₁, . . . , β₅ using logisticregression on the training set of shots S. The target for the regressionis 0 when the shot is missed and 1 when the shot is made. By performingtwo regressions, one is able to find appropriate values for thecoefficients, for both shots within 10 feet, and longer shots outside 10feet.

As depicted in FIG. 23, three or four dimensions can be dynamicallydisplayed on a 2-D graph scatter rank view 2302, including the x, y,size of the icon, and changes over time. Each dimension may be selectedby the user to represent a variable of the user's choice. Also, onmouse-over, related icons may highlight, e.g., moussing over one playermay highlight all players on the same team.

As depicted in FIG. 40, reports 2402 can be customized by the user sothat a team can create a report that is specifically tailored to thatteam's process and workflow. Another feature is that the report mayvisually display not only the advantages and disadvantages for eachcategory shown, but also the size of that advantage or disadvantage,along with the value and rank of each side being compared. This visuallanguage enables a user to quickly scan the report and understanding themost important points.

Referring to FIG. 25, an embodiment of a quality assurance UI 2502 isprovided. The QA UI 2502 presents the human operator with both ananimated 2D overhead view 2510 of the play, as well as a video clip 2508of the play. A key feature is that only the few seconds relevant to thatplay are shown to the operator, instead of an entire possession, whichmight be over 20 seconds long, or even worse, requiring the humanoperator to fast forward in the game tape to find the event herself.Keyboard shortcuts are used for all operations, to maximize efficiency.Referring to FIG. 26, the operator's task is simplified to its core, sothat we lighten the cognitive load as much as possible: if the operatoris verifying a category of plays X, the operator has to simply choose,in an interface element 2604 of the embodiment of the QA UI 2602 whetherthe play shown in the view 2608 is valid (Yes or No), or (Maybe). Shecan also deem the play to be a (Duplicate), a (Compound) play that meansit is just one type-X action in a consecutive sequence of type-Xactions, or choose to (Flag) the play for supervisor review for anyreason. Features of the UI 2602 include the ability to fast word,rewind, submit and the like, as reflected in the menu element 2612. Atable 2610 can allow a user to indicate the validity of plays occurringat designated times.

FIG. 27 shows a method of camera pose detection, also known as “courtsolving.” FIG. 27 also shows the result of automatic detection of the“paint,” and use of the boundary lines to solve for the camera pose. Thecourt lines and hoop location, given the solved camera pose, are thenshown projected back onto the original image 2702. This projection isfrom the first iteration of the solving process, and one can see thatthe projected court and the actual court do not yet align perfectly. Onemay use machine vision techniques to find the hoop and to find the courtlines (e.g., paint boundaries), then use found lines to solve for thecamera pose. Multiple techniques may be used to determine court lines,including detecting the paint area. Paint area detection can be doneautomatically. One method involves automatically removing the non-paintarea of the court by automatically executing a series of “flood fill”type actions across the image, selecting for court-colored pixels. Thisleaves the paint area in the image, and it is then straightforward tofind the lines/points. One may also detect all lines on the court thatare visible, e.g., background or 3-point arc. In either case,intersections provide points for camera solving. A human interface 2702may be provided for providing points or lines to assist algorithms, tofine-tune the automatic solver. Once all inputs are provided, the camerapose solver is essentially a randomized hill climber that uses themathematical models as a guide (since it may be under-constrained). Itmay use multiple random initializations. It may advance a solution if itis one of the best in that round. When an iteration is done, it mayrepeat until the error is small. FIG. 46 shows the result of automaticdetection of the “paint,” and use of the boundary lines to solve for thecamera pose. The court lines and hoop location, given the solved camerapose, are then shown projected back onto the original image. Thisprojection is from the first iteration of the solving process, and onecan see that the projected court and the actual court do not yet alignperfectly.

FIG. 28 relates to camera pose detection. The second step 2802 shown inthe Figure shows how the human can use this GUI to manually refinecamera solutions that remain slightly off.

FIG. 29 relates to auto-rotoscoping. Rotoscoping 2902 is required inorder to paint graphics around players without overlapping the players'bodies. Rotoscoping is partially automated by selecting out the parts ofthe image with similar color as the court. Masses of color left in theimage can be detected to be human silhouettes. The patch of color can be“vectorized” by finding a small number of vectors that surround thepatch, but without capturing too many pixels that might not represent aplayer's body.

FIGS. 30A, 30B, and 30C relate to scripted storytelling with an assetlibrary 3002. To produce the graphics-augmented clips, a company mayeither lean heavily on a team of artists, or a company may determine howbest to handle scripting based on a library of assets. For example,instead of manually tracing a player's trajectory and increasing theshot probability in each frame as the player gets closer to the ball, ascripting language allows the methods and systems described herein tospecify this augmentation in a few lines of code. In another example,for rebound clips, the Voronoi partition and the associated reboundpositioning percentages can be difficult to compute for every frame. Alibrary of story element effects may list each of these current andfuture effects. Certain combinations of scripted story element effectsmay be best suited for certain types of clips. For example, a reboundand put-back will likely make use of the original shot probability, therebound probabilities including Voronoi partitioning, and then go backto the shot probability of the player going for the rebound. This entirescript can be learned as being well-associated with the event type inthe video. Over time, the system can automatically infer the best, or atleast retrieve an appropriate, story line to match up with a selectedvideo clip containing certain events. This enables augmented videoclips, referred to herein as DataFX clips, to be auto-generated anddelivered throughout a game.

FIGS. 31-38 show examples of DataFX visualizations. The visualization ofFIG. 31 requires court position to be solved in order to lay down grid,player “puddles”. Shot arc also requires backboard/hoop solution. InFIG. 32, Voronoi tessellation, heat map, shot and rebound arcs allrequire the camera pose solution. The highlight of the player usesrotoscoping. In FIG. 33, in addition to the above, players arerotoscoped for highlighting. FIGS. 34-38 show additional visualizationsthat are based on the use of the methods and systems disclosed herein.

In embodiments, DataFX (video augmented with data-driven specialeffects) may be provided for pre-, during, or post-game viewing, foranalytic and entertainment purposes. DataFX may combine advanced datawith Hollywood-style special effects. Pure numbers can be boring, whilepure special effects can be silly, but the combination of the two andthe results can be very powerful. Example features used alone or incombination in DataFX can include use of a Voronoi overlay on court, aGrid overlay on court, a Heat map overlay on court, a Waterfall effectshowing likely trajectories of the ball after a missed field goalattempt, a Spray effect on a shot, showing likely trajectories of theshot to the hoop, Circles and glows around highlighted players,Statistics and visual cues over or around players, Arrows and othermarkings denoting play actions, Calculation overlays on court, andeffects showing each variable taken into account.

FIGS. 39A through 41B show a product referred to as “Clippertron.”Provided is a method and system whereby fans can use their distributedmobile devices to control individually and/or collectively what is shownon the Jumbotron or video board(s). An embodiment enables the fan to gothrough mobile application dialogs in order to choose the player, shottype, and shot location to be shown on the video board. The fan can alsoenter in his or her own name so that it is displayed alongside thehighlight clip. Clips are shown on the Video Board in real time orqueued up for display. Variations include getting information about thefan's seat number. This could be used to show a live video feed of thefan while their selected highlight is being shown on the video board.Referred to as “FanMix” is a web-based mobile application that enablesin-stadium fans to control the Jumbotron and choose highlight clips topush to the Jumbotron. An embodiment of FanMix enables fans to choosetheir favorite player, shot type, and shot location using a mobiledevice web interface. Upon pressing the submit button, a highlightshowing this particular shot is sent to the Jumbotron and displayedaccording to placement order in a queue. Enabling this capability isthat video is lined up to each shot within a fraction of a second. Thisallows many clips to be shown in quick succession, each showing videofrom the moment of release to the ball going through the hoop. In somecases, the video may start from the beginning of a play, instead of whena play begins.

The methods and systems disclosed herein may include methods and systemsfor allowing a user or group of users to control presentation of a largescale display in an event venue, where the options for control are basedon a context of the content as determined by machine extraction ofsemantically relevant events from the content

The methods and systems disclosed herein may include methods and systemsfor enabling interaction with a large scale display system and mayinclude taking a video feed; using machine learning to develop anunderstanding of an event within the video feed, the understandingincluding identifying context information relating to the event;automatically, under computer control, extracting the content displayingthe event and associating the extracted content with the contextinformation; producing a video content data structure that includes theassociated context information; and providing an application by which atleast one user can interact with the video content data structure,wherein the options for user interaction are based on the contextinformation, wherein the interaction with the video content datastructure controls the presentation of the content on a large scaledisplay.

In embodiments, one or more users may interact with menus on anapplication, such as a smart phone application, in an arena or otherlocation that has a large-scale display. The users may expresspreferences, such as by voting, for what content should be displayed,including selecting preferred types of events and/or contexts (which maybe organized as noted above based on semantically relevant filters),selecting what metrics should be displayed (options for which may beoffered based on context information for particular extracted videoevents), and the like. In embodiments, a large scale display in a venuewhere a live event is taking place may offer games, quizzes, or thelike, where users may respond by text, SMS, or the like. The content ofsuch games or quizzes may be constructed at least in part based on amachine semantic understanding of the live event, such as asking userswhich player has the most rebounds in the first quarter, or the like.

The methods and systems disclosed herein may include methods and systemsfor a user to control Jumbotron clips based on contextualized contentfilters.

The methods and systems disclosed herein may include methods and systemsfor a Jumbotron fan quiz based on machine semantic understanding of alive game

The methods and systems disclosed herein may include methods and systemswherein the application comprises a quiz for a user, wherein the quiz isconstructed based at least in part on a machine semantic understandingof a live game that is taking place in a venue where the large-scaledisplay is located. In embodiments, a fan quiz may ask questions basedon proprietary machine learned metrics such as “which player took thehardest shots in this quarter.” The methods and systems disclosed hereinmay include methods and systems for embedding a machine extracted videocut in an application, where the selection of the embedded cut for theapplication is based on the context of the video cut.

First Person Point of View (POV)

In embodiments, interactive visualization 218, as illustrated in FIG. 2,may include producing a reconstruction of an event, such as a game, suchas a 3D reconstruction or rendering. In embodiments, a 3D reconstructionor rendering of an event may be produced using a process that presentsthe event from a defined point of view, such as the first person pointof view of a participant in the event, such as a player. FIG. 39Fillustrates an embodiment of such as process, referred to herein in somecases as a first person POV process, or simply a first person process.

A first person process may allow the user to select a player's view tofollow. A first person process may automatically pin a user's view tothe head of the selected player. The end result of a first personprocess may be dynamically rendered from the view of the selected playeras a play occurs.

A first person process may be an automated first person process. Anautomated first person process may produce a 3D reconstruction orrendering of a game and render each frame from the view of a playerselected by a user.

A first person process may be a virtual reality-based first personprocess. A virtual reality-based first person process may produce a 3Dreconstruction or rendering of a game that allows a user to control theorientation of a view from the head movements of a user. In embodiments,the point of view may be controlled by, for example, player headtracking.

In embodiments, users may choose a player whose point of view will bepresented. Location of a view may be controlled automatically via headtracking data. View orientation may be controlled by the head movementsof a user. In embodiments, the head movements of a user may be recordedby virtual reality (VR) technology. VR technology may be Oculus Rift™technology and the like.

Point Cloud Construction

As illustrated in FIG. 39F, a first person process may includeconstructing a point cloud that provides a 3D model of a real worldscene.

Point cloud construction may begin by producing binary,background-subtracted images for each time-synchronized frame on eachcamera. Using these binary images and the calibrations of each camera, a3D convex hull may be produced by discretizing the scene into voxels andfilling each voxel, if the voxel is contained within the ray projectedfrom the camera through the image visual hull. The image visual hull maybe the silhouette of the scene, for example. The silhouette of the scenemay be a shape-form silhouette.

The resulting convex hull may contain voxels that may not actually bepresent in the world, due to reconstructing only of the visual hull. Inorder to achieve a more precise point cloud, the 3D convex hull may becarved using photo consistency methods.

Photo consistency methods may back-project the surface of a 3Dreconstructed visual hull onto each visible camera. Photo consistencymethods may also check to ensure the color of the pixels is consistentwith the same pixel from another camera, or with nearby pixels, such asto avoid unrealistic discontinuities. If the colors from each visiblecamera do not agree, the voxel may be carved. This process may berepeated for the entire convex hull, producing the final carved pointcloud.

Point cloud construction may estimate the skeletal pose of allparticipants in a real world scene. Point cloud construction may fit ahand-made participant model to the estimated pose of each participant ina real world scene. In an example, the real world scene could be asports court and the participants could be all the players on the sportscourt. In this example, point cloud construction could fit a hand-madeplayer model to the estimated pose of each player on the sports court.

Point cloud construction may include meshing techniques, which may beused to improve the quality of a final visualization for a user. Meshingtechniques may be used to mesh multiple point counts. Meshing techniquesmay be used to provide a view that may be very close to a point cloud,for example.

Player Identification

A first person process may use player identification to enable the userto select from which player's view to render the 3D reconstruction.Player identification may involve multiple steps in order to producereliable results.

Player identification may start by performing jersey number detection,as illustrated in FIG. 39F. Jersey numbers may be mapped to playernames. Jersey numbers may then be mapped to player names using officialrosters and the like.

Jersey number detection may be performed frame-by-frame. Frame-by-framejersey number detection may be performed by scanning and classifyingeach window as a number or as nothing, such as using a support vectormachine (SVM), a supervised machine learning model used forclassification. The SVM may be trained, such as using training sets ofmanually marked jersey numbers from the game video, for example.

Results from individual frame-by-frame detection may be stitchedtogether to form temporal tracks. Individual frame by frame detectionmay be stitched together to form temporal tracks using a k-shortestpaths algorithm, as illustrated in FIG. Y. Jersey number tracks may beassociated with existing, more continuous player tracking data.Associating jersey number tracks with existing, more continuous playertracking data may produce robust tracks of identifiable players.

Head Tracking

A first person process may use head tracking in order to control thelocation of the view within a 3D reconstruction, as illustrated in FIG.39F. Head tracking may involve multiple steps in order to producereliable results.

The first step of head tracking may be the same as for playeridentification. The first step of head tracking may include headdetection. Head detection may create a model on heads instead of onjersey numbers. Head detection may be performed frame by frame.

Head detection may include frame by frame head detection. Frame-by-framehead detection may be performed by scanning each image. Frame-by-framehead detection may be performed by scanning each image and classifyingeach window as a head or not.

Classifying each window as a head or not may be performed using an SVM.An SVM may be trained. An SVM may be trained using manually marked headsamples from previously recorded games. An SVM maybe be a team-dk-SVM.

The results of the detection may then be used in 2D tracking to producetemporal 2D tracklets of each head within a camera's frame. 2D trackletsmay then be triangulated using the results of all cameras to produce a3D estimation of the location of all heads on the court. A 3D estimationof the location of all heads on the court may be 3D tracklets.

3D tracklets may then be stitched together. 3D tracklets may then bestitched together using an algorithm. An algorithm may be a k-shortestpaths (KSP) algorithm. 3D tracklets may be stitched together to producepotential final head tracking results. Linear programming may be used tochoose optimal head paths.

Gaze Estimation

As illustrated in FIG. 39F, a first person process may use gazeestimation. Gaze estimation may be used to control the orientation of aview mounted on the player's head within the 3D reconstruction. Gazeestimation may be computed by assuming a player is always looking in thedirection opposite the numbers on the back of the player.

Jersey number detection may be performed frame by frame. Frame by framejersey number detection may be performed by scanning and classifyingeach window as a number or nothing using an SVM. The SVM may be trainedusing manually marked jersey numbers from an existing game video.

An assumption may be made to determine the angle of a jersey numberlocated on the back or front of a player's jersey. An assumption may bethat a jersey number is only visible when the jersey number is perfectlyaligned with a camera that made the detection.

Cameras may have a known location in space. Because the cameras have aknown location in space, the vector between the jersey and the cameramay be computed using the known location of the camera in space.

Frame-by-frame estimation may be performed after a vector is calculated.The results of the frame-by-frame estimation may be filtered to providea smoothed experience for a first person process.

FIG. 41 relates to an offering referred to as “inSight.” This offeringallows pushing of relevant stats to fans' mobile devices 4104. Forexample, if player X just made a three-point shot from the wing, thiswould show statistics about how often he made those types of shots 4108,versus other types of shots, and what types of play actions he typicallymade these shots off of inSight does for hardcore fans what Eagle (thesystem described above) does for team analysts and coaches. Information,insights, and intelligence may be delivered to fans' mobile deviceswhile they are seated in the arena. This data is not only beautiful andentertaining, but is also tuned into the action on the court. Forexample, after a seemingly improbable corner three by a power forward,the fan is immediately pushed information that shows the shot'sfrequency, difficulty, and likelihood of being made. In embodiments, theplatform features described above as “Eagle,” or a subset thereof may beprovided, such as in a mobile phone form factor for the fan. Anembodiment may include a storyboard stripped down, such as from a formatfor an 82″ touch screen to a small 4″ screen. Content may be pushed to adevice that corresponds to the real time events happening in the game.Fans may be provided access to various effects (e.g., DataFX featuresdescribed herein) and to the other features of the methods and systemsdisclosed herein.

FIGS. 42 and 43 show touchscreen product interface elements 4202, 4204,4208, 4302 and 4304. These are essentially many different skins anddesigns on the same basic functionality described throughout thisdisclosure. Advanced stats are shown in an intuitive large-format touchscreen interface. A touchscreen may act as a storyboard for showingvarious visualizations, metric and effects that conform to anunderstanding of a game or element thereof. Embodiments include a largeformat touch screen for commentators to use during a broadcast. WhileInSight serves up content to a fan, the Storyboard enables commentatorson TV to access content in a way that helps them tell the mostcompelling story to audiences.

Features include providing a court view, a hexagonalFrequency+Efficiency View, a “City/Matrix” View with grids of events, aFace/Histogram View, Animated intro sequences that communicate to aviewer that each head's position means that player's relative ranking,an Animated face shuttle that shows re-ranking when metric is switched,a ScatterRank View, a ranking using two variables (one on each axis), aTrends View, integration of metrics with on-demand video and the abilityto r-skin or simplify for varying levels of commentator ability.

In embodiments, new metrics can be used for other activities, such asdriving new types of fantasy games, e.g., point scoring in fantasyleagues could be based on new metrics.

In embodiments, DataFX can show the player how his points were scored,e.g., overlay that runs a counter over an RB's head showing yards rushedwhile the video shows RB going down the field. In embodiments, one candeliver, for example, video clips (possibly enhanced by DataFX effects)corresponding to plays that scored points for a fantasy user's team forthat night or week.

Using an inSight-like mobile interface, a social game can be made sothat much of the game play occurs in real time while the fan is watchingthe game.

Using Insight-like mobile device features, a social game can be managedso that game play occurs in real time while a fan is watching the game,experiencing various DataFX effects and seeing fantasy scoring-relevantmetrics on screen during the game. In embodiments, the methods andsystems may include a fantasy advice or drafting tool for fans,presenting rankings and other metrics that aid in player selection.

Just as Eagle enables teams to get more wins by devising better tacticsand strategy, we could provide an Eagle-like service for fantasy playersthat gives the players a winning edge. The service/tool would enablefans to research all the possible players, and help them execute abetter draft or select a better lineup for an upcoming week/game.

DataFX can also be used for instant replays with DataFX optimized sothat it can produce “instant replays” with DataFX overlays. This relieson a completely automated solution for court detection, camera posesolving, player tracking, and player roto-scoping.

Interactive DataFX may also be adapted for display on a second screen,such as a tablet, while a user watches a main screen. Real time orinstant replay viewing and interaction may be used to enable sucheffects. On a second screen-type viewing experience, the fan couldinteractively toggle on and off various elements of DataFX. This enablesthe fan to customize the experience and to explore many differentmetrics. Rather than only DataFX-enabled replays, the system could befurther optimized so that DataFX is overlaid in true real time, enablingthe user to toggle between a live video feed and a live video feed thatis overlaid with DataFX. The user would then also be able to choose thetype of DataFX to overlay, or which player(s) to overlay it on.

A touch screen UI may be established for interaction with DataFX.

Many of the above embodiments may be used for basketball, as well as forother sports and for other items that are captured in video, such as TVshows, movies, or live video (e.g., news feeds). For sports, a playertracking data layer may be employed to enable the computer to“understand” every second of every game. This enables the computer todeliver content that is extracting from portions of the game and toaugment that content with relevant story-telling elements. The computerthus delivers personalized interactive augmented experiences to the enduser.

For non-sports domains, such as TV shows or movies, there is no playertracking data layer that assists the computer in understanding theevent. Rather, in this case, the computer must derive, in some otherway, an understanding of each scene in a TV show or movie. For example,the computer might use speech recognition to extract the dialoguethroughout a show. Or it could use computer vision to recognize objectsin each scene, such as robots in the Transformer movie. Or it could usea combination of these inputs and others to recognize things likeexplosions. The sound track could also provide clues.

The resulting system would use this understanding to deliver the samekind of personalized interactive augmented experience as we havedescribed for the sports domain. For example, a user could request tosee the Transformer movie series, but only a compilation of the sceneswhere there are robots fighting and no human dialogue. This enables“short form binge watching,” where users can watch content created bychopping up and recombining bits of content from original video. Theoriginal video could be sporting events, other events TV shows, movies,and other sources. Users can thus gorge on video compilations thattarget their individual preferences. This also enables a summary form ofwatching, suitable for catching up with current events or currentlytrending video, without having to watch entire episodes or movies.

FIG. 44 provides a flow under which the platform may ingest and alignthe content of one or more broadcast video feeds and one or moretracking camera video feeds. At a step 4412, a broadcast video feed maybe ingested, which may consist of an un-calibrated and un-synchronizedvideo feed. The ingested broadcast video feed may be processed byperforming optical character recognition at a step 4414, such as toextract information from the broadcast video feed that may assist withaligning events within the feed with events identified in other sourcesof video for the same event. This may include recognizing text andnumerical elements in the broadcast video feed, such as game scores, thegame clock, player numbers, player names, text feeds displayed on thevideo, and the like. For example, the time on the game clock, or thescore of a game, may assist with time-alignment of a broadcast feed withanother video feed. At a step 4404 objects may be detected within thebroadcast video feed 4404, such as using machine-basedobject-recognition technologies. Objects may include players (includingbased on recognizing player numbers), body parts of players (e.g., headsof players, torsos of players, etc.) equipment (such as the ball in abasketball game), and many others. Once detected at the step 4404,objects may be tracked over time in a step 4418, such as in progressiveframes of the broadcast video feed. Tracked objects may be used toassist in calibrating the broadcast video intrinsic and extrinsic cameraparameters by associating the tracked objects with the same objects asidentified in another source, such as a tracking camera video feed.Abc123

At a step 4402, in parallel with the steps involved in ingesting andprocessing a broadcast video feed, video feeds from tracking cameras,such as tracking cameras for capturing 3D motion in a venue (like asports arena), may be ingested. The tracking camera video feeds may becalibrated and synchronized to a frame of reference, such as one definedby the locations of a set of cameras that are disposed at knownlocations within the venue where the tracking camera system ispositioned. At a step 4406, one or more objects may be detected withinthe tracking camera video feed, including various objects of the typesnoted above, such as players, numbers, items of equipment, and the like.In embodiments, spatiotemporal coordinates of the objects may bedetermined by processing the information from the tracking camera videofeed, the coordinates being determined for the recognized objects basedon the frame of reference defined by the camera positions of thetracking system. In embodiments, the coordinates being determined forthe recognized objects can be based on the court or the field on whichthe game is played. In embodiments, the coordinates being determined forthe recognized objects are based on the boundaries, lines, markers,indications, and the like associated with the court or the field onwhich the game is played. The video feed from the tracking camera systemand the information about spatiotemporal object positions may be used togenerate a point cloud at a step 4416, within which voxel locations ofthe objects detected at the step 4406 may be identified at a step 4418.The tracking camera video feed that was processed to detect and trackobjects may be further processed at a step 4410 by using spatiotemporalpattern recognition (such as machine-based spatiotemporal patternrecognition as described throughout this disclosure) to identify one ormore events, which may be a wide range of events as described throughoutthis disclosure, such as events that correspond to patterns in a game orsport.

In embodiments, other feeds may be available that may contain additionalinformation about events that are contained in the tracking camera videofeed. For example, a data feed, such as a play-by-play feed, for a gamemay be ingested at a step 4422. At a step 4420, the information frommultiple sources may be aligned, such as aligning the play-by-play datafeed from the step 4422 with events recognized at the step 4410.Similarly, at a step 4424 the recognized event data in the trackingcamera video feed at the step 4410 may be aligned with events recognizedin the broadcast video feed at the step 4414, resulting in time-alignedbroadcast video, tracking camera, and other (e.g., play-by-play) feeds.Once the tracking camera video feed and the broadcast video feed aretime-aligned for an event, objects detected at the step 4404 in thebroadcast video feed and tracked at the step 4418 (e.g., players' heads)may be used at a step 4428 to calibrate the broadcast video cameraposition, such as by identifying the broadcast video camera positionwithin the frame of reference of the tracking camera system used tocapture the tracking camera video feed. This may include comparing sizesand orientations of the same object as it was detected at the step 4404in the broadcast video feed and at the step 4406 in the tracking camerasystem video feed. In embodiments, calibration parameters of thebroadcast camera can be determined by, among other things, comparingpositions of detected objects in the video with detectedthree-dimensional positions of the corresponding objects that can beobtained using the calibrated tracking system. In embodiments, heads ofthe players in the game can be suitable objects because the heads of theplayers can be precisely located relative to other portions of thebodies of the players. Once calibrated, the broadcast video camerainformation can be processed as another source just like any of thetracking cameras. This may include re-calibrating the broadcast videocamera position for each of a series of subsequent events, as thebroadcast video camera may move or change zoom between events. Once thebroadcast video camera position is calibrated to the frame of referenceof the tracking camera system, at a step 4430 pixel locations in thebroadcast video feed may be identified, corresponding to objects in thebroadcast video feed, which may include using information about voxellocations of objects in the point cloud generated from the motiontracking camera feed at the step 4418 and/or using image segmentationtechniques on the broadcast video feed. The process of FIG. 44 thusprovides time-aligned broadcast video feeds, tracking camera eventfeeds, and play-by-play feeds, where within each feed pixel locations orvoxel locations of objects and backgrounds are known, so that variousactivities can be undertaken to process the feeds, such as foraugmenting the feeds, performing pattern recognition on objects andevents within them (such as to find plays following particularpatterns), automatically clipping or cutting them to produce content(such as capturing a reaction in broadcast video to an event displayedin or detected by the tracking camera feeds based on a time sequence oftime-aligned events), and many others as described throughout thisdisclosure.

In some embodiments, the platform may use stationary features on aplaying surface (e.g., a basketball court) to calibrate the broadcastvideo camera parameters and to time align two or more video feeds. Forexample, the platform may utilize stationary lines (e.g., yard lines,top of the three point line, a half court line, a center field line,side lines, intersections between half court or field lines and sidelines, logos, goal posts, and the like) to calibrate the broadcast videocamera parameters. In these embodiments, the stationary features may bedetected in the broadcast video feed and in the tracking video feed. Inembodiments, the platform may determine the x, y, and z locations of thestationary features in the tracking video feed, and may calibrate thebroadcast video camera parameters based on the x, y, z coordinates ofthe stationary features or voxel coordinates. For example, inembodiments, the platform may cross-reference the pixel locations of astationary feature in the broadcast video feed with the x, y, zcoordinates of the stationary feature in the tracking camera feeds. Oncethe broadcast video feed is calibrated with respect to one or moretracking camera feeds, moving objects tracked in the broadcast video canbe cross-referenced against the locations of the respective movingobjects from the tracking camera video feeds. In some of theseembodiments, the platform may track moving objects in the broadcastvideo feed and the tracking camera feed(s) with respect to the locationsof the stationary features in the respective broadcast video feed andtracking camera feeds to time align the broadcast video feed andtracking camera feeds. For example, the platform may time align one ormore broadcast video feeds and one or more tracking camera feeds atrespective time slices where a player crosses a logo or other stationaryfeatures on the playing surface in each of the respective feeds(broadcast video and tracking camera feeds).

Referring to FIG. 45, embodiments of the methods and systems disclosedherein may involve handling multiple video feeds 4502, information fromone or more tracking systems 4512 (such as player tracking systems thatmay provide time-stamped location data and other information, such asphysiological monitoring information, activity type information, etc.),and one or more other input sources 4510 (such as sources of audioinformation, play-by-play information, statistical information, eventinformation, etc.). In embodiments, live video input feeds 4502 areencoded by one or more encoding systems 4504 to produce a series ofvideo segment files 4508, each consisting of a video chunk, optionallyof short duration, e.g., four seconds. Video segment files 4508 fromdifferent input feeds corresponding to the same time interval areconsidered as part of a temporal group 4522 associated with that timeinterval. The temporal group 4522 may also include information and othercontent from tracking systems 4512 and other input sources 4510.

In embodiments, each video segment file 4508 may independently and inparallel undergo various processing operations in one or more processingsystems 4518, such as transcoding to various file formats, streamingprotocols, and the like. The derived output files 4520 of processing4518 may be associated with the same temporal group 4522. Temporalgrouping 4522 enables time synchronization among the original andderived files without having to further maintain or track timing orsynchronization information. Such processing operations 4518 mayinclude, without limitation, standard video on demand (VOD) transcoding,such as into lower bit rate video files. Processing operations 4518 mayalso include augmentation, such as with graphics, audio overlays, ordata, producing derived, augmented video files 4518. Other data derivedfrom the video streams or obtained from other sources 4510 (e.g.,coordinate positions of players and objects obtained via optical or chiptracking systems 4512), which may typically become available with asmall time delay relative to the live video input streams 4502, may alsobe synchronized to the video files 4508 in a temporal group 4522, suchas by adding them as metadata files to the corresponding temporal groupor by binding them to the video files 4514. In embodiments, a manifestfile 4524 based on these temporal groups 4522 may be created to enablestreaming of the original video feed 4502, the video segment files 4514and/or derived files 4520 as a live, delayed or on-demand stream.Synchronization among the output streams may enable combining and/orswitching 4528 seamlessly among alternative video feeds (e.g., differentangles, encoding, augmentations or the like) and data feeds of a livestreamed event.

Among other benefits, synchronization across original video feeds 4502,video segment files 4508, derived video feeds 4520 with encoded,augmented or otherwise processed content, and backup video feeds,described by a manifest file 4524, may allow client-side failover fromone stream to another without time discontinuity in the viewing of theevent. For instance, if an augmented video stream resulting fromprocessing 4518 is temporarily unavailable within the time offset atwhich the live stream is being viewed or falls below a specifiedbuffering amount, a client application 4530 consuming the video feed maytemporarily fail over to an un-augmented stream 4502 or encoded videosegment file 4508.

In embodiments, the granularity with which the client application 4530switches back to the augmented stream 4518 when available may depend onsemantically defined boundaries in the video feed, which in embodimentsmay be based on a semantic understanding of events within the videofeed, such as achieved by the various methods and systems described inconnection with the technology stack 100 and the processes 200 describedthroughout this disclosure. For example, a switch back to derived file4520 with various augmentations added in processing 4518 may be timed tooccur after a change of possession, a timeout, a change in camera angle,a change in point-of-view, or other appropriate points in the action, sothat the switching occurs while minimizing disruption of the viewingexperience. Switching may also be controlled by semantic understanding4532 of the content of different video feeds 4502 at each time instant;for example, if a camera is not pointing at the current action on thecourt, an alternative video feed 4502, video segment file 4514 orderived file 4520 may be selected.

In embodiments, a “smart pipe” may be provided consisting of multiplealigned content channels (e.g., audio, video, or data channels) that areindexed both temporally and spatially. Spatial indexing and alignment4534 may include indexing of pixels in 2D streams, voxels in 3D streams,and other objects, such as polygonal meshes used for animation, 3Drepresentation, or the like. In embodiments, a wide variety of elementsmay be indexed, such as, without limitation, events, and locations ofobjects (including players, game objects, and objects in theenvironment, such as a court or arena) involved in those events. Inembodiments, a further variety of elements may be indexed includinginformation and statistics related to events and locations. Inembodiments, a further variety of elements may be indexed includinglocations of areas corresponding to floor areas, background areas,signage areas, or the like where information, augmentations, graphics,animations, advertising, or the like may be displayed over a contentframe. In embodiments, a further variety of elements may be indexedincluding indices or indicators of what information, augmentationelements or the like that are available to augment a video feed in acontent channel such as ones that may be selected individually or incombination.

In embodiments, a further variety of elements may be indexed includingpredefined combinations of content (e.g., particular combinations ofaudio, video, information, augmentation elements, replays, or othercontent elements), such as constituting channels or variations fromwhich end-users may choose ones that they prefer. Thus, a spatialindexing and alignment system 4534 may provide spatial indexing andalignment information to the processing system 4518 (or may be includedtherein), such that the derived files 4520 (and optionally variousobjects therein) that are indexed both temporally and spatially. In sucha case, the “smart pipe” for synchronized, switchable and combinablecontent streams 4528 may contain sufficient indexed and aligned contentto allow the creation of derived content, the creation of interactiveapplications, and the like, each optionally tied to live and recordedevents (such as sporting events). In embodiments, the tracking systems4512, the spatial indexing and alignment 4534 and the semanticunderstanding 4532 may be part of the larger alignment, tracking, andsemantic system included in the systems and methods disclosure hereinthat may take various inputs including original video feeds andplay-by-play feeds, and may produce X, Y, Z tracking data and semanticlabels. The X, Y, Z tracking data and semantic labels may be stored asseparate metadata files in the temporal group 4522 or used to producederived video files 4520 in the temporal group 4522.

In embodiments, any combination of inputs such as from a tracking camerasystem, a 3D camera array, broadcast video, a smartphone video, lidar,and the like may be used to automatically obtain a 3D understanding of agame. The automatically obtained 3D understanding of the game may beused to index voxels of 3D representations (e.g., AR/VR video) or pixelsof any 2D video footage (e.g., from tracking cameras, broadcast,smartphones, reconstructed video from any point of view such as firstperson point of view of players in the game) or alternatively tovoxels/pixels, other graphics representations such as polygonal meshes.

In embodiments, a “smart pipe” may consist of multiple aligned contentchannels (e.g., audio, video, or data channels) that are indexed bothtemporally and spatially (e.g., indexing of pixels/voxels/polygonalmeshes) with events and locations of players/objects involved in thoseevents. By way of this example, the indexing both temporally andspatially with events and locations of players/objects involved in thoseevents may also include information and statistics related to events andlocations. The indexing both temporally and spatially with events andlocations of players/objects involved in those events may also includelocations of areas corresponding to floor or background areas whereinformation, augmentations (e.g., filters that manipulate the look ofthe ball/players) or advertising may be displayed over each video frame.In embodiments, available pieces of information and augmentationelements may be selected individually or in combination. In embodiments,combinations of audio, video, information, augmentation, replays, andthe like may constitute channels for end-users to choose from. The smartpipe may contain sufficient indexed and aligned content to createderived content and interactive apps tied to live and recorded games.

In embodiments, composition of video via frames, layers and/or tracksmay be generated interactively by distributed sources, e.g., base videoof the sporting event, augmentation/information layers/frames fromdifferent providers, audio tracks from alternative providers,advertising layers/frames from other providers, leveraging indexing andsynchronization concepts, and the like. By way of this example, the baselayers and/or tracks may be streamed to the various providers as well asto the clients. In embodiments, additional layers and/or tracks may bestreamed directly from the providers to the clients and combined at theclient. In embodiments, the composition of video via frames, layersand/or tracks and combinations thereof may be generated interactively bydistributed sources and may be based on user personalizations.

In embodiments, the systems and methods described herein may include asoftware development kit (SDK) 4804 that enables content being played ata client 4808 to dynamically incorporate data or content from at leastone separate content feed 4802. In these embodiments, the SDK 4804 mayuse timecodes or other timing information in the video to align theclient's current video playout time with data or content from the atleast one separate content feed 4802, in order to supply the videoplayer with relevant synchronized media content 4810.

In operation, a system 4800 (e.g., the system described herein) mayoutput one or more content feeds 4802-1, 4802-2 . . . 4802-N. Thecontent feeds may include video, audio, text, and/or data (e.g.,statistics of a game, player names). In some embodiments, the system4800 may output a first content feed 4802-1 that includes a video and/oraudio that is to be output (e.g., displayed) by a client media player4808. The client media player 4808 may be executed by a user device(e.g., a mobile device, a personal computing device, a tablet computingdevice, and the like). The client media player 4808 is configured toreceive the first content feed 4802 and to output the content feed 4802via a user interface (e.g., display device and/or speakers) of the userdevice. Additionally or alternatively, the client media player 4808 mayreceive a third-party content feed 4812 from a third-party data source(not shown). For example, the client media player 4808 may receive alive-game video stream from the operator of an arena. Regardless of thesource, a content feed 4802-2 or 4812 may include timestamps or othersuitable temporal indicia to identify different positions (e.g., framesor chunks) in the content feed. The client media player 4808 mayincorporate the SDK 4804. The SDK 4804 may be configured to receiveadditional content feeds 4802-2 . . . 4802-N to supplement the outputtedmedia content. For example, a content feed 4802-2 may include additionalvideo (e.g., a highlight or alternative camera angle). In anotherexample, a content feed 4802-2 may include data (e.g., statistics orcommentary relating to particular game events). Each additional contentfeed 4802-2 . . . 4802-N may include timestamps or other suitabletemporal indicia as well. The SDK 4804 may receive the additionalcontent feed(s) 4802-2 . . . 4802-N and may augment the content feedbeing output by the media player with the one or more additional contentfeeds 4802-2 . . . 4802-N based on the timestamps of the respectivecontent feeds 4802-1, 4802-2, . . . 4802-N to obtain dynamicsynchronized media content 4810. For example, while playing a live feed(with a slight lag) or a video-on-demand (VOD) feed of a basketballgame, the SDK 4804 may receive a first additional content feed 4802containing a graphical augmentation of a dunk in the game and a secondadditional content feed 4802 indicating the statistics of the player whoperformed the dunk. The SDK 4804 may incorporate the additional contentfeeds into the synchronized media content 4810, by augmenting the dunkin the live or VOD feed with the graphical augmentation and thestatistics. In some embodiments, a client app using the SDK may allowclient-side selection or modification of which subset of the availableadditional content feeds to incorporate. In some implementations, theSDK 4804 may include one or more templates that define a manner by whichthe different content feeds 4802 may be laid out. Furthermore, the SDK4804 may include instructions that define a manner by which theadditional content feeds 4802 are to be synchronized with the originalcontent feed.

In embodiments, the systems and methods disclosed herein may includejoint compression of channel streams such as successive refinementsource coding to reduce streaming bandwidth and/or reduce channelswitching time, and the like.

In embodiments, the systems and methods disclosed herein may includeevent analytics and/or location-based games including meta-games,quizzes, fantasy league and sport, betting, and other gaming optionsthat may be interactive with many of the users at and connected to theevent such as identity-based user input, e.g., touching or clicking aplayer predicted to score next. In embodiments, the event analyticsand/or location-based games may include location-based user input suchas touching or clicking a location where a rebound or other play oractivity is expected to be caught, to be executed, and the like. Inembodiments, the event analytics and/or location-based games may includetiming-based user input such clicking or pressing a key to indicate whena user thinks a shot should be taken, a defensive play should beinitiated, a time-out should be requested, and the like. In embodiments,the event analytics and/or location-based games may includeprediction-based scoring including generating or contributing to a userscore based on the accuracy of an outcome prediction associated with theuser. By way of this example, the outcome prediction may be associatedwith outcomes of individual offensive and defensive plays in the gamesand/or may be associated with scoring and/or individual playerstatistics at predetermined time intervals (e.g., quarters, halves,whole games, portions of seasons, and the like). In embodiments, theevent analytics and/or location-based games may include game state-basedscoring including generating or contributing to a user score based onexpected value of user decision calculated using analysis ofinstantaneous game state and/or comparison with evolution of game statesuch as maximum value or realized value of the game state in a givenchance or possession.

In embodiments, the systems and methods disclosed herein may includeinteractive and immersive reality games based on actual game replays. Byway of this example, the interactive and immersive reality games mayinclude the use of one or more simulations to diverge from actual gameevents (partially or in their entirety) based on user input or acollection of user input. In embodiments, the interactive and immersivereality games may include an action-time resolution engine that may beconfigured to determine a plausible sequence of events to rejoin theactual game timeline relative to, in some examples, the one or moresimulations to diverge from actual game events (partially or in theirentirety) based on user input or a collection of user input. Inembodiments, the interactive and immersive reality games may includeaugmented reality simulations that may integrate game event sequences,using cameras on located on one or more backboards and/or alonglocations adjacent to the playing court. In embodiments, the systems andmethods disclosed herein may include simulated sports games that may bebased on detailed player behavior models. By way of this example, thedetailed player behavior models may include tendencies to take differentactions and associated probabilities of success of different actionsunder different scenarios including teammate/opponent identities,locations, score differential, period number, game clock, shot clock,and the like.

In embodiments, the systems and methods disclosed herein may includesocial chat functions and social comment functions that may be insertedinto a three-dimensional scene of a live event. By way of this example,the social chat and comment functions that may be inserted into thethree-dimensional scene of the live event may include avatars insertedinto the crowd that may display comments within speech bubbles above theavatars. In other examples, the social chat and comment functions may beinserted into a three-dimensional scene of the live event as a runningcommentary adjacent to other graphics or legends associated with theevent.

In embodiments, the systems and methods disclosed herein may include theautomating of elements of broadcast production such as automatic controlof camera pan, tilt, and zoom. By way of this example, the automating ofelements of broadcast production may also include automatic switchingbetween camera views. In embodiments, the automating of elements ofbroadcast production may include automatic live and color commentarygeneration and automatic placement and content from syntheticcommentators in the form of audio or in the form of one or more audioand video avatars with audio content that may be mixed with semantic andcontextual based reactions from the live event and/or from other users.By way of this example, the automated elements of broadcast productionmay include automated generation of commentary in audio only or audioand video form including AR augmentation and associated content by, forexample, combining semantic machine understanding of events in the gameand semantic machine understanding of camera views, camera cuts, andcamera close-ups in broadcast or another video.

In embodiments, the automated generation of commentary may also be basedon semantic machine understanding of broadcaster/game audio, statisticsfrom semantic machine understanding of past games,information/statistics from other sources, and combinations thereof. Inembodiments, a ranking of potential content items may be based on atleast one of the rarity of events, comparison against the rest of theleague, diversity with respect to previously shown content,personalization based on channel characteristics, explicit userpreferences, inferred user preferences, the like, or combinationsthereof. In embodiments, the automated generation of commentary mayinclude the automatic selection of top-ranked content items or a shortlist of top-ranked content items shown to a human operator forselection.

In embodiments, and as shown in FIG. 49, the systems and methodsdisclosed herein may include machine-automated or machine-assistedgeneration of aggregated clips 4902. Examples of aggregated clips 4902include highlights and/or condensed games. The aggregated clip may becomprised of one or more selected media segments (e.g., video and/oraudio segments). In the example of FIG. 49, a multimedia system 4900 mayinclude an event datastore 4910, an interest determination module 4920,and a clip generation module 4930. The event datastore 4910 may storeevent records 4912. Each event records 4912 may correspond to arespective event (e.g., an offensive possession, a shot, a dunk, adefensive play, a blitz, a touchdown pass). An event record 4912 mayinclude an event ID 4914 that uniquely identifies the event. An eventrecord 4912 may also include event data 4916 that corresponds to theevent. For example, event data 4916 may include a media segment (e.g.,video and/or audio) that captures the event or a memory address thatpoints to the media segment that captures the event. The event record4912 may further include event metadata 4918. Event metadata 4918 mayinclude any data that is pertinent to the event. Examples of eventmetadata 4918 may include, but is not limited to, an event type (e.g., abasketball shot, a dunk, a football blitz, a touchdown, a soccer goal),a list of relevant players (e.g., the shooter and defender, thequarterback, the goal scorer), a time corresponding to the event (e.g.,when during the game did the event occur), a length of the event (e.g.,how many seconds is the media segment that captures the event), asemantic understanding of the event, the potential impact event on winprobability (e.g., a delta of win probability from before and after theevent), references (e.g., event IDs) to other events that are pertinentto event (e.g., other events during a run made by a team, and/or anyother suitable types of metadata. In some embodiments, the eventmetadata 4918 may further include an interest score of the event, wherethe interest score of an event may be a numerical value indicating adegree of likelihood that a user would find the event interesting (e.g.,worthy of watching).

In embodiments, an interest determination module 4920 determines aninterest level of an event or group of related events. In some of theseembodiments, the interest determination module 4920 determines aninterest score of an event or group of related events. The interestscore may be relative to other events in a particular game or relativeto events spanning multiple games and/or sports. In some embodiments,the interest determination module 4920 may determine the interest scoreof a particular event or group of events based on the event metadata4918 of the respective event(s). In some embodiments, the interestdetermination module 4920 may incorporate one or more machine-learnedmodels that receive event metadata 4918 of an event or group of relatedevents and outputs a score based on the event metadata 4918. Amachine-learned model may, for example, receive an event type, and otherrelevant features (e.g., time, impact on win probability, relevantplayer) and may determine the score based thereon. The machine-learnedmodels may be trained in a supervised, semi-supervised manner, orunsupervised manner. The interest determination module 4920 maydetermine the interest score of an event or group of related events inother manners as well. For example, the interest determination module4920 may utilize rules-based scoring techniques to score an event orgroup of related events.

In some embodiments, the interest determination module 4920 isconfigured to determine an interest score for a particular user. Inthese embodiments, the interest scores may be used to generatepersonalized aggregated clips 4902 for a user. In these embodiments, theinterest determination module 4920 may receive user-specific data thatmay be indicative of a user's personal biases. For example, the interestdetermination module 4920 may receive user-specific data that mayinclude, but is not limited to, a user's favorite sport, the user'sfavorite team, the user's list of favorite players, a list of eventsrecently watched by the user, a list of events recently skipped by theuser, and the like. In some of these embodiments, the interestdetermination module 4920 may feed the user-specific data intomachine-learned models along with event metadata 4818 of an event todetermine an interest score that is specific to a particular user. Inthese embodiments, the interest determination module 4920 may output theuser-specific interest score to the clip generation module 4930.

In some embodiments, one or more humans may assign interest levels tovarious events. In these embodiments, the human-assigned interest levelsmay be used to determine which events to include in an aggregated clip4902. Furthermore, the human-assigned interest levels may be used totrain a model used to determine interest scores of respective events.

The clip generation module 4930 generates aggregated clips 4902 based onone or more identified events. The clip generation module 4930 maydetermine one or more events to include in an aggregated clip based onthe interest level of the events relating to a game or collection ofgames. In some embodiments, the clip generation module 4930 determinesthe events to include in an aggregated clip 4902 based on the interestlevel of the respective events. The clip generation module 4930 mayimplement optimization or reinforcement learning to determine whichevents (depicted in media segments) to include in an aggregated clip4902. For instance, the clip generation module 4930 may include mediasegments depicting events having the highest relative interest scoresand media segments of additional events that may be relevant to the highscoring events. In embodiments, the clip generation module 4930 maydetermine how many events to include in the aggregated clip 4902depending on the intended purpose of the aggregated clip 4902. Forexample, a highlight may be shorter in duration than a condensed game.In embodiments, the length of an aggregated clip 4902 may be apredetermined parameter (e.g., three minutes). In these embodiments, theclip generation module 4930 may select a sufficient number of events tospan the predetermined duration. For example, the clip generation module4930 may identify a set of media segments of events having requisiteinterest scores, where the aggregated duration of the set of mediasegments is approximately equal to the predetermined duration.

In embodiments, the clip generation module 4930 may be configured togenerate personalized aggregated clips. In these embodiments, the clipgeneration module 4930 may receive user-specific interest scorescorresponding to events of a particular game or time period (e.g.,“today's personalized highlights). The clip generation module 4930 mayutilize the user-specific interest scores of the events, a user'shistory (e.g., videos watched or skipped), and/or user profile data(e.g., location, favorite teams, favorite sports, favorite players) todetermine which events to include in a personalized aggregated clip4930. In embodiments, the clip generation module 4930 may determine howmany events to include in the personalized aggregated clip 4902depending on the intended purpose of the aggregated clip 4902 and/or thepreferences of the user. For example, if a user prefers to have longercondensed games (i.e., more events in the aggregated clip), the clipgeneration module 4930 may include more media segments in the aggregatedclip. In some embodiments, the length of an aggregated clip 4902 may bea predetermined parameter (e.g., three minutes) that may be explicitlyset by the user. In these embodiments, the clip generation module 4930may select a sufficient number of events to span the predeterminedduration set by the user. For example, the clip generation module 4930may identify a set of media segments of events having requisite interestscores, where the aggregated duration of the set of media segments isapproximately equal to the predetermined duration.

In embodiments, the clip generation module 4930 requests the scores ofone or more events from the interest determination module 4920 when theclip generation module 4930 is tasked with generating aggregated clips4902. Alternatively, the interest determination module 4920 may scoreeach event defined in the event datastore 4910. Upon determining whichevents to include in an aggregated clip 4902, the clip generation module4930 may retrieve the media segments corresponding to the identifiedevents. For example, the clip generation module 4930 may retrieve theevent records 4912 of the identified events using the event IDs 4914 ofthe identified events. The clip generation module 4914 may then generatethe aggregated clip based on the event data 4916 contained in theretrieved event records 4912. The sequence of events depicted in theaggregated clip 4902 may be generated in any suitable manner. Forexample, the events may be depicted sequentially as they occurred or inorder of ascending or descending interest score. The clip generationmodule 4930 may transmit the aggregated clip 4902 to a user deviceand/or store the aggregated clip 4902 in memory.

In embodiments, and in the example of FIG. 50, the systems and methodsdisclosed herein may be configured to provide “dynamic videos” 5002. Adynamic video 5002 may refer to the concatenated display of mediasegments (e.g., video and/or audio) that can be dynamically selectedwith short time granularity (e.g., frame-level or chunk-levelgranularity). A dynamic video 5002 may be comprised of one or moreconstituent media segments of dynamically determined length, content,and sequencing. The dynamic video 5002 may include constituent mediasegments that are stitched together in a single file or a collection ofseparate files that may each contain a respective constituent mediasegment. The constituent media segments of a dynamic video 5002 may berelated based on one or more suitable relationships. For example, theconstituent media segments may be of a same event taken from differentcamera angles, of different events of a same game, of different eventsfrom different games but of the same sport and on the same day, ofdifferent events relating to the same player or team, and/or ofdifferent events but the same subject, topic, or sentiment.Additionally, in some embodiments, the constituent media segments may besupplemented or augmented with graphical and/or text overlays. Thegraphical and/or text overlays may be confined to a single media segmentor may span across multiple constituent media segments.

In the illustrated example, a multimedia system 5000 provides thedynamic videos 5002 to a user device 5080. The user device 5080 may be amobile device (e.g., smartphone), a personal digital assistant, a laptopcomputing device, a personal computer, a tablet computing device, agaming device, a smart television, and/or any other suitable electronicdevice with the capability to present the dynamic videos. The userdevice 5080 may include a multimedia player 5082 that outputs thedynamic video 5002 via a user interface 5084. The multimedia player 5082may also receive user commands via the user interface 5084. The userinterface 5084 may include a display device (e.g., an LED screen or atouchscreen), a physical keyboard (e.g., a qwerty keyboard), an inputdevice (e.g., a mouse), an audio device (e.g., speakers), and the like.The user device 5080 may further include a communication unit 5088 thateffectuates communication with external devices directly and/or via anetwork. For example, the communication unit 5088 may include one ormore wireless and/or wired transceivers that communicate using anysuitable communication protocol.

The multimedia system 5000 may include a media datastore 5010, acommunication unit 5030, and a dynamic video module 5020. The mediadatastore 5010 may store media records 5012. A media record 5012 maycorrespond to a media segment that captures one or more events. A mediarecord may include a media ID 5014 that uniquely identifies the mediarecord 5012. A media record 5012 may include media data 5016. The mediadata 5016 may include the media segment itself or a memory address ofthe media segment. The media record 5012 may further include mediametadata 5018. The media metadata 5018 may include any data that ispertinent to the media segment. Examples of media metadata 4918 mayinclude, but is not limited to, one or more event identifiers theidentify one or more events depicted in the media segment, one or moreevent types that describe the one or more events depicted in the mediasegment, a list of relevant players depicted in the multimedia segment,a time corresponding to the media segment (e.g., a starting time of themedia segment with respect to a game), a time length of the mediasegment, a semantic understanding of the media segment, the potentialimpact of the events depicted in the media segment on win probability(e.g., a delta of win probability from before and after the event),references (e.g., media IDs) to other media segments that are pertinentto the media segment (e.g., other angles of the same events depicted inthe media segment), and/or any other suitable types of metadata. Inembodiments, the media records 5012 may further reference entire contentfeeds (e.g., an entire game or a livestream of a game). In theseembodiments, the media metadata 5018 of a media record may include anysuitable information relating to the content feed. For example, themedia metadata 5018 may include an identifier of the game to which thecontent feed corresponds, an indicator whether the content feed is liveor recorded, identifiers of the teams playing in the game, identifiersof players playing in the game, and the like.

The dynamic video module 5020 is configured to generate dynamic videosand to deliver dynamic videos to a user device 5080. The dynamic videomodule 5020 may select the media segments to include in the dynamicvideo 5002 in any suitable manner. In some embodiments, the dynamicvideo module 5020 may implement optimization and/or reinforcementlearning-based approaches to determine the selection, length, and/orsequence of the constituent media segments. In these embodiments, thedynamic video module 5020 may utilize the media metadata 5018 of themedia records 5012 stored in the media datastore 5010 to determine theselection, length, and/or sequence of the constituent media segments.The dynamic video module 5020 may additionally or alternativelyimplement a rules based approach to determine which media segments toinclude in the dynamic video. For example, the dynamic video module 5020may be configured to always include alternative camera angles of anevent if multiple media segments depicting the same event exist. In thisexample, the dynamic video module 5020 may be further configured todesignate media clips taken from alternative camera angles assupplementary media segments (i.e., media segments that can be switchedto at the user device) rather than sequential media segments. Inembodiments, the dynamic video module 5020 may be configured to generatedynamic video clips from any suitable sources, including content feeds.In these embodiments, the dynamic video module 5020 may generate dynamicvideos 5002 having any variety of constituent media segments by cuttingmedia segments from one or more content feeds and/or previously cutmedia segments. Furthermore, the dynamic video module 5020 may add anycombination of augmentations, graphics, audio, statistics, text, and thelike to the dynamic video.

In some embodiments, the dynamic video module 5020 is configured toprovide personalized dynamic videos 5002. The dynamic video module 5020may utilize user preferences (either predicted, indicated, or inferred)to customize the dynamic video. The dynamic video 5030 may utilize auser's profile, location, and/or history to determine the userpreferences. A user profile may indicate a user's favorite teams,players, sports, and the like. In another example, the dynamic videomodule 5020 may be able to predict a user's favorite teams and playersbased on the location of the user. In yet another example, the dynamicvideo module 5020 may be configured to infer user viewing preferencesbased on the viewing history of the user (e.g., telemetry data reportedby the media player of the user). For example, if the user historyindicates that the user routinely skips over media segments that arelonger than 30 seconds, the dynamic video module 5020 may infer that theuser prefers media segments that are less than 30 seconds long. Inanother example, the dynamic video module 5020 may determine that theuser typically “shares” media segments that include reactions of playersor spectators to a notable play. In this example, the dynamic videomodule 5020 may infer that the user prefers videos that includereactions of players or spectators, and therefore, media segments thattend to be longer in duration. In another example, the user history mayindicate that the user watches media segments of a particular type ofevent (e.g., dunks), but skips over other types of events (e.g., blockedshots). In this example, the dynamic video module 5020 may infer thatthe user prefers to consume media segments of dunks over media segmentsof blocked shots. In operation, the dynamic video module 5020 canutilize the indicated, predicted, and/or inferred user preferences todetermine which media segments to include in the dynamic video and/orthe duration of the media segments (e.g., should the media segment beshorter or longer). The dynamic video module 5020 may utilize anoptimization and/or reinforcement-based learning approach to determinewhich media segments to include in the dynamic video 5002, the durationof the dynamic video 5002, and the sequence of the media segments in thedynamic video 5002.

The multimedia system 5000 may transmit a dynamic video 5002 to a userdevice 5080. The media player 5082 receives the dynamic video 5002 viathe communication unit 5088 and outputs one or more of the mediasegments contained in the dynamic video 5002 via the user interface5084. The media player 5082 may be configured to record user telemetrydata (e.g., which media segments the user consumers, which mediasegments the user skips, and/or terms that the user searches for) and toreport the telemetry data to the multimedia system 5000. The mediaplayer 5084 may be configured to receive commands from a user via theuser interface 5084. The commands may be executed locally by the mediaplayer 5084 and/or may be communicated to the multimedia system 5000.

In some embodiments, the media player 5082 may be configured to allowselection of the media segments that are displayed based on user inputand/or AI-controls. In the former scenario, the media player 5082 may beconfigured to receive user commands via the user interface 5084. Forexample, the media player 5082 may allow a user to enter search terms orto choose from a displayed set of suggestions. In response to the searchterms or the user selections, the media player 5082 may initialize(e.g., request and begin outputting) a dynamic video 5002, in which themedia player 5082 displays a machine-controlled sequence of mediasegments related to the search terms/user selection. A user may issueadditional commands via the user interface 5084 (e.g., via the keyboardor by touching or directional swiping on a touchscreen) to request mediasegments related in different ways to the current media segment, toindicate when to move on to the next media segment, and/or tointeractively pull up statistics and other information. For example,swiping upwards may indicate that the user wishes to see a differentcamera angle of the same event, swiping downwards may indicate that theuser wishes to see an augmented replay of the same event, and swipingright may indicate that the user wishes to move on to the next clip. Aset of keyword tags corresponding to each clip may be shown tofacilitate the user adding one or more of the displayed tags to the setof search terms that determines potentially relevant media segments todisplay. The media player 5082 may report the user's inputs orinteractions with the media player 5082, if any, to the multimediasystem 5000. In response to such commands, the multimedia system 500 mayuse such data to adapt subsequent machine-controlled choices of mediasegment duration, content type, and/or sequencing in the dynamic video.For example, the user's inputs or interactions may be used to adjust theparameters and/or reinforcement signals of an optimization orreinforcement learning-based approach for making machine-controlledchoices in the dynamic video 5002.

In embodiments, the dynamic video module 5020 may be configured togenerate the dynamic video in real time. In these embodiments, thedynamic video module 5020 may begin generating and transmitting thedynamic video 5002. During display of the dynamic video 5002 by themedia player 5082, the dynamic video module 5020 may determine how tosequence/curate the dynamic video. For instance, the dynamic videomodule 5020 may determine (either based on a machine-learning-baseddecision or from explicit instruction from the user) that the angle of alive feed should be switched to a different angle. In this situation,the dynamic video module 5020 may update the dynamic video 5002 with adifferent video feed that is taken from an alternative angle. In anotherexample, a user may indicate (either explicitly or implicitly) that sheis uninterested in a type of video being shown (e.g., baseballhighlights). In response to the determination that the user isuninterested, the dynamic video module 5020 may retrieve media segmentsrelating to another topic (e.g., basketball) and may begin stitchingthose media segments into the dynamic video 5002. In this example, thedynamic video module 5020 may be configured to cut out any mediasegments that are no longer relevant (e.g., additional baseballhighlights). It is noted that in some embodiments, the dynamic videomodule 5020 may transmit alternative content feeds and/or media segmentsin the dynamic video 5002. In these embodiments, the media player 5082may be configured to switch between feeds and/or media segments.

In embodiments, the automating of elements of broadcast production mayinclude automatic live commentary generation that may be used to assistreferees for in situ evaluation or post-mortem evaluation. The automaticlive commentary generation that may be used to assist referees may alsobe used to train referees in unusual situations that may be seeninfrequently in actual games but may be reproduced or formed from ARcontent based on or purposefully deviated from live game events. By wayof the above examples, the referee assistance, evaluation, training, andthe like associated with the improvement of referee decisions may bebased on semantic machine understanding of game events.

In embodiments, the systems and methods disclosed herein may include theuse of player-specific information in three-dimensional positionidentification and reconstruction to improve trade-offs among camerarequirements. Toward that end, fewer or lower resolution cameras may beused, computational complexity/delay may be reduced and outputquality/accuracy may be increased when compared to typical methods. Withreference to FIG. 46, the player-specific information inthree-dimensional position identification and reconstruction 4600 may beshown to improve the balance in trade-offs of camera requirementsincluding improved localization of keypoints 4602 such as a head,joints, and the like, by using player models 4604 of specific players inconjunction with player identification 4608 such as identifying a jerseynumber or automatically recognizing a face and remote sensing technologyto capture the players such as one or more video cameras, lidar,ultrasound, Wi-Fi visualization, and the like. By way of this example,the improved localization of keypoints may include optimizing overconstraints on distances between keypoints from player models combinedwith triangulation measurements from multiple cameras.

In embodiments, the improved localization of keypoints may also includeusing the player models 4604 to enable 3D localization with a singlecamera. In embodiments, the system and methods disclosed herein may alsoinclude the use of the player models 4604 fitted to detected keypointsto create 3D reconstructions 4620 or to improve 3D reconstructions incombination with point cloud techniques. Point cloud techniques mayinclude a hybrid system including the player models 4604 that may beused to replace areas where the point cloud reconstruction does notconform adequately to the model. In further examples, the point cloudtechniques may include supplementing the point cloud in scenarios wherethe point cloud may have a low density of points. In embodiments, theimproved localization of keypoints may include the use of player heightinformation combined with face detection, gaze detection, posturedetection, or the like to locate the point of view of players.

In embodiments, the improved localization of keypoints may also includethe use of camera calibration 4630 receiving one or more video feeds4632, the 3D reconstruction 4610 and projection onto video in order toimprove player segmentation for broadcast video 4640.

In embodiments, the systems and methods disclosed herein may includeusing a state-based machine learning model with hierarchical states. Byway of this example, the state-based machine learning model withhierarchical states may include input training state labels at thefinest granularity. In embodiments, the machine learning model may betrained at the finest level of granularity as well as at intermediatelevels of aggregated states. In embodiments, the output and costfunction optimization may be at the highest level of state aggregation.In embodiments, the machine learning model may be trained using anensemble of active learning methods for multiclass classificationincluding weighting of methods based on a confusion matrix and a costfunction that may be used to optimize the distribution of qualitativelyvaried instances for active learning.

FIG. 51 illustrates an example of a client device 5100 configured todisplay augmented content to a user according to some embodiments of thepresent disclosure. In the illustrated example, the client device 5100may include a processing device 5102, a storage device 5104, acommunication unit 5106 that effectuates communication between theclient device and other devices via one or more communication networks(e.g., the Internet and/or a cellular network), and a user interface5108 (e.g., a touchscreen, a monitor, a mouse, a keyboard, and thelike). The processing device 5102 may include one or more processors andmemory that stores computer-executable instructions that are executed bythe one or more processors. The processing device 5102 may execute avideo player application 5200. In embodiments, the video playerapplication 5200 is configured to allow a user to consume video andrelated content from different content channels (e.g., audio, video,and/or data channels). In some of the embodiments, the video and relatedcontent may be delivered in time-aligned content channels (e.g., a“smart pipe”), where the content may be indexed temporally and/orspatially. In embodiments, the spatial indexing may include indexing thepixels or groups of pixels of multiple streams, 3D pixels (e.g., voxels)or groups of 3D pixels, and/or objects (e.g., polygonal meshes used foranimation, overlay graphics, and the like). In these embodiments, a widevariety of elements may be indexed temporally (e.g., in relation toindividual video frames) and/or spatially (e.g., in relation to pixels,groups of pixels, or “real world” locations depicted in the videoframes). Examples of elements that may be indexed include events(match/game identifier), objects (players, game objects, objects in theenvironment such as court or playing field) involved in an event,information and statistics relating to the event and locations,locations of areas corresponding to the environment (e.g., floor areas,background areas, signage areas) where information, augmentations,graphics, animations, and advertising can be displayed in a frame,indicia of what information, augmentation elements, and the like thatare available to augment a video feed in a content channel, combinationsof content (e.g., particular combinations of audio, video, information,augmentation elements, replays, or other suitable elements), and/orreferences to other content channels corresponding to the event (suchthat end-users can select between streams). In this way, the videoplayer may allow a user to interact with the video, such that the usercan request the video player to display information relating to a timeand/or location in the video feed, display relevant information relatingto the event, switch between video feeds of the event, viewadvertisements, and the like. In these embodiments, the smart pipe mayallow the video player 5200 to create dynamic content at the clientdevice 5100.

FIG. 52 illustrates an example implementation of the video playerapplication 5200 according to some embodiments of the presentdisclosure. The video player application 5200 may include a GUI module5202, an integration module 5204, an access management module 5206, avideo transformation module 5208, a time transformation module 5210, anda data management module 5212. The video player application 5200 mayinclude additional or alternative modules not discussed herein withoutdeparting from the scope of the disclosure.

In embodiments, the GUI module 5202 receives commands from a user anddisplays video content, including augmented video content, to the uservia the user interface 5108. In embodiments, the GUI module 5202displays a menu/selection screen (e.g., drop down menus, selectionelements, and/or search bars) and receives commands from a usercorresponding to the available menus/selection items via a user via theuser interface 5108. For example, the GUI module 5202 may receive anevent selection via a drop down menu and/or a search bar/results page.In embodiments, an event selection may be indicative of a particularsport and/or a particular match. In response to an event selection, theGUI module 5202 may provide the event selection to the integrationmodule 5204. In response, the GUI module 5202 may receive a video stream(of one or more video streams capturing the selected event) from thevideo transformation module 5208 and may output a video corresponding tothe video feed via the user interface 5112. The GUI module 5202 mayallow a user to provide commands with respect to the video content,including commands such as pause, fast forward, and rewind. The GUImodule 5202 may receive additional or alternative commands, such as“make a clip,” drill down commands (e.g., provide stats with respect toa player, display players on the playing surface, show statisticscorresponding to a particular location, and the like), switch feedcommands (e.g., switch to a different viewing angle), zoom in/zoom outcommands, select link commands (e.g., selection of an advertisement),and the like.

The integration module 5204 receives an initial user command to view aparticular sport or game and instantiates an instance of a video player(also referred to as a “video player instance”). In embodiments, theintegration module 5204 receives a source event identifier (ID), anaccess token, and/or a domain ID. The source event ID may indicate aparticular game (e.g., MLB: Detroit Tigers v. Houston Astros). Theaccess token may indicate a particular level of access that a user haswith respect to a game or league (e.g., the user may access advancedcontent or MLB games may include multi-view feed). The domain ID mayindicate a league or type of event (e.g., NBA, NFL, FIFA). Inembodiments, the integration module may instantiate a video playerinstance in response to the source event ID, the domain ID, and theaccess token. The integration module 5204 may output the video playerinstance to the access management module 5206. In some embodiments, theintegration module 5204 may further output a time indicator to theaccess management module 5206. A time indicator may be indicative of atime corresponding to a particular frame or group of frames within thevideo content. In some of these embodiments, the time indicator may be awall time. Other time indicators, such as a relative stream (e.g., 10seconds from t=0), may be used, however.

The access management module 5206 receives the video player instance andmanages security and/or access to video content and/or data by the videoplayer from a multimedia system. In embodiments, the access managementmodule 5206 may expose a top layer API to facilitate the ease of accessto data by the video player instance. The access management module 5206may determine the level of access to provide the video player instancebased on the access token. In embodiments, the access management module5206 implements a single exported SDK that allows a data source (e.g.,multimedia servers) to manage access to data. In other embodiments, theaccess management module 5206 implements one or more customized exportedSDKs that each contain respective modules for interacting with arespective data source. The access management module 5206 may be a passthrough layer, whereby the video player instance is passed to the videotransformation module 5208.

The video transformation module 5208 receives the video player instanceand obtains video feeds and/or additional content provided by amultimedia server (or analogous device) that may be displayed with thevideo encoded in the video feeds. In embodiments, the videotransformation module 5208 receives the video content and/or additionalcontent from the data management module 5212. In some of theseembodiments, the video transformation module 5208 may receive a smartpipe that contains one or more video feeds, audio feeds, data feeds,and/or an index. In embodiments, the video feeds may be time-alignedvideo feeds, such that the video feeds offer different viewing angles orperspectives of the event to be displayed. In embodiments, the index maybe a spatio-temporal index. In these embodiments, the spatio-temporalindex identifies information associated with particular video frames ofa video and/or particular locations depicted in the video frames. Insome of these embodiments, the locations may be locations in relation toa playing surface (e.g., at the fifty yard line or at the free throwline) or defined in relation to individual pixels or groups of pixels.It is noted that the pixels may be two-dimensional pixels orthree-dimensional pixels (e.g., voxels). The spatio-temporal index mayindex participants on a playing surface (e.g., players on a basketballcourt), statistics relating to the participants (e.g., Player A hasscored 32 points), statistics relating to a location on the playingsurface (e.g., Team A has made 30% of three-pointers from a particulararea on a basketball court), advertisements, score bugs, graphics, andthe like. In some embodiments, the spatio-temporal index may index walltimes corresponding to various frames. For example, the spatio-temporalindex may indicate a respective wall time for each video frame in avideo feed (e.g., a real time at which the frame was captured/initiallystreamed).

The video transformation module 5208 receives the video feeds and theindex and may output a video to the GUI module 5202. In embodiments, thevideo transformation module 5208 is configured to generate augmentedvideo content and/or switch between different video feeds of the sameevent (e.g., different camera angles of the event). In embodiments, thevideo transformation module 5208 may overlay one or more GUI elementsthat receive user selections into the video being output. For example,the video transformation module 5208 may overlay one or more visualselection elements over the video feed currently being output by the GUImodule 5202. The visual selection elements may allow a user to viewinformation relating to the event depicted in the video feed, to switchviews, or to view a recent highlight. In response to the user providinga command via the user interface of the client device 5100, the videotransformation module 5208 may augment the currently displayed videofeed with augmentation content, switch the video feed to another videofeed, or perform other video transformation related operations.

The video transformation module 5208 may receive a command to displayaugmentation content. For example, the video transformation module 5208may receive a command to display information corresponding to aparticular location (e.g., a pixel or group of pixels) and a particularframe. In response to the command, the video transformation module 5208may reference the spatio-temporal index to determine an object (e.g., aplayer) that is located at the particular location in the particularframe. The video transformation module 5208 may retrieve informationrelating to the object. For example, the video transformation module5208 may retrieve a name of a player or statistics relating to a playeror a location on the playing surface. The video transformation module5208 may augment the current video feed with the retrieved content. Inembodiments, the video transformation module 5208 may request thecontent (e.g., information) from the multimedia server via the datamanagement module 5212. In other embodiments, the content may betransmitted in a data feed with the video feeds and the spatio-temporalindex. In response to receiving the requested content (which may betextual or graphical), the video transformation module 5208 may overlaythe requested content on the output video. The video transformationmodule 5208 may determine a location in each frame at which to displaythe requested data. In embodiments, the video transformation module 5208may utilize the index to determine a location at which the requestedcontent may be displayed, whereby the index may define locations in eachframe where specific types of content may be displayed. In response todetermining the location at which the requested content may bedisplayed, the video transformation module 5208 may overlay the contentonto the video at the determined location.

In another example, the video transformation module 5208 may receive acommand to display an advertisement corresponding to a particular frameand location. In response to the command, the video transformationmodule 5208 determines the advertisement to display from thespatio-temporal index based on the particular frame and location. Inembodiments, the video transformation module 5208 may retrieve theadvertisement from the multimedia server (or another device). In otherembodiments, the advertisement may be transmitted with the video feedsand the spatio-temporal index. In response to obtaining theadvertisement, the video transformation module 5208 may determine alocation at which the advertisement is to be displayed (e.g., in themanner discussed above), and may overlay the advertisement onto thevideo at the determined location.

In embodiments, the video transformation module 5208 may receive acommand to switch between video feeds in response to a user command toswitch feeds. In response to such a command, the video transformationmodule 5208 switches the video feed from the current video feed to arequested video feed, while maintaining time-alignment between the video(i.e., the video continues at the same point in time but from adifferent feed). For example, in streaming a particular basketball gameand receiving a request to change views, the video transformation module5208 may switch from a sideline view to an under the basket view withoutinterrupting the action of the game. The video transformation module5208 may time align the video feeds (i.e., the current video feed andthe video feed being switched to) in any suitable manner. In someembodiments, the video transformation module 5208 obtains a wall timefrom the time transformation module 5210 corresponding to a currentframe or upcoming frame. The video transformation module 5208 mayprovide a frame identifier of the current frame or the upcoming frame tothe video transformation module 5208. In embodiments, the frameidentifier may be represented in block plus offset form (e.g., a blockidentifier and a number of frames within the block). In response to theframe identifier, the time transformation module 5208 may return a walltime corresponding to the frame identifier. The video transformationmodule 5208 may switch to the requested video feed, whereby the videotransformation module 5208 begins playback at a frame corresponding tothe received wall time. In these embodiments, the video transformationmodule 5208 may obtain the wall time corresponding to the current orupcoming frame from the time transformation module 5210, and may obtaina frame identifier of a corresponding frame in the video feed beingswitched to based on the received wall time. In some embodiments, thevideo transformation module 5208 may obtain a “block plus offset” of aframe in the video feed being switched to based on the wall time. Theblock plus offset may identify a particular frame within a video streamas a block identifier of a particular video frame and an offsetindicating a number of frames into the block where the particular videoframe is sequenced. In some of these embodiments, the videotransformation module 5208 may provide the video transformation module5210 with the wall time and an identifier of the video feed beingswitched, and may receive a frame identifier in block plus offset formatfrom the time transformation module 5210. In some embodiments, the videotransformation module 5208 may reference the index using a frameidentifier of a current or upcoming frame in the current video feed todetermine a time aligned video frame in the requested video feed. It isnoted that while the “block plus offset” format is described, otherformats of frame identifiers may be used without departing from thescope of the disclosure. In response to obtaining a frame identifier,the video transformation module 5208 may switch to the requested videofeed at the determined time aligned video frame. For example, the videotransformation module 5208 may queue up the requested video feed at thedetermined frame identifier. The video transformation module 5208 maythen begin outputting video corresponding to the requested video feed atthe determined frame identifier.

In embodiments, the time transformation module 5210 receives an inputtime value in a first format and returns an output time value in asecond format. For example, the time transformation module 5210 mayreceive a frame indicator in a particular format (e.g., block plusoffset”) that indicates a particular frame of a particular video feed(e.g., the currently displayed video feed of an event) and may return awall time corresponding to the frame identifier (e.g., the time at whichthe particular frame was captured or was initially broadcast). Inanother example, the time transformation module 5210 receives a walltime indicating a particular time in a broadcast and a request for aframe identifier of a particular video feed. In response to the walltime and the frame identifier request, the time transformation module5210 determines a frame identifier of a particular video frame within aparticular video feed and may output the frame identifier in response tothe request. The time transformation module 5210 may determine theoutput time in response to the input time in any suitable manner. Inembodiments, the time transformation module 5210 may utilize an indexcorresponding to an event (e.g., the spatio-temporal index correspondingto an event) to determine a wall time in response to a frame identifierand/or a frame identifier in response to a wall time. In theseembodiments, the spatio-temporal index may be keyed by frame identifiersand/or wall times, whereby the spatio-temporal index returns a wall timein response to a frame identifier and/or a frame identifier in responseto a wall time and a video feed identifier. In other embodiments, thetime transformation module 5210 calculates a wall time in response to aframe identifier and/or a frame identifier in response to a wall time.In some of these embodiments, each video feed may include metadata thatincludes a starting wall time that indicates a wall time at which therespective video feed began being captured/broadcast, a number of framesper block, and a frame rate of the encoding. In these embodiments, thetime transformation module 5210 may calculate a wall time in response toa frame identifier based on the starting time of the video feedindicated by the frame identifier, the number of frames per block, andthe frame indicated by the frame identifier (e.g., the block identifierand the offset value). Similarly, the time transformation module 5210may calculate a frame identifier of a requested video feed in responseto a wall time based on the starting time of the requested video feed,the received wall time, the number of frames per block, and the encodingrate.

In some embodiments, the time transformation module 5210 may beconfigured to transform a time with respect to first video feed to atime with respect to a second video feed. For example, the timetransformation module 5210 may receive a first frame indicatorcorresponding to a first video feed and may output a second frameindicator corresponding to a second video feed, where the first frameindicator and the second frame indicator respectively indicatetime-aligned video frames. In some of these embodiments, the timetransformation module 5210 may utilize an index corresponding to anevent (e.g., the spatio-temporal index corresponding to an event) todetermine the second frame identifier in response to the second frameidentifier. In these embodiments, the spatio-temporal index may be keyedby frame identifiers and may index frame identifiers of video framesthat are time-aligned with the video frame referenced by each respectiveframe identifier. In other embodiments, the time transformation module5210 calculates the second frame identifier in response to the firstidentifier. In some of these embodiments, the time transformation module5210 may convert the first frame identifier to a wall time, as discussedabove, and then may calculate the second frame identifier based on thewall time, as described above.

In embodiments, the data management module 5212 requests and/or receivesdata from external resources and provides the data to a requestingmodule. For example, the data management module 5212 may receive the oneor more video feeds from a multimedia server. The data management module5212 may further receive an index (e.g., spatio-temporal index)corresponding to an event being streamed. For example, in someembodiments, the data management module 5212 may receive a smart pipecorresponding to an event. The data management module 5212 may providethe one or more video feeds and the index to the video transformationmodule 5208. In embodiments, the data management module 5212 may exposeone or more APIs of the video player application to external resources,such multimedia servers and/or related data servers (e.g., a server thatprovides game information such as player names, statistics, and thelike). In some embodiments, the external resources may push data to thedata management module 5212. Additionally or alternatively, the datamanagement module 5212 may be configured to pull the data from theexternal resources.

In embodiments, the data management 5212 may receive requests for datafrom the video management module 5208. For example, the data managementmodule 5212 may receive a request for information relating to aparticular frame identifier, a location within the frame indicated by aframe identifier, and/or an object depicted in the frame indicated by aframe identifier. In these embodiments, the data management module 5212may obtain the requested information and may return the requestedinformation to the video management module 5212. In some embodiments,the external resource may push any information that is relevant to anevent to the data management module 5212. In these embodiments, the datamanagement module 5212 may obtain the requested data from the pusheddata. In other embodiments, the data management module 5212 may beconfigured to pull any requested data from the external resource. Inthese embodiments, the data management module 5212 may transmit arequest to the external resource, whereby the request indicates theinformation sought. For example, the request may indicate a particularframe identifier, a location within the frame indicated by a frameidentifier, or an object (e.g., a player) depicted in the frameindicated by the frame identifier. In response to the request, the datamanagement module 5212 may receive the requested information, which ispassed to video transformation module 5212.

In embodiments, the data management module 5212 may be configured toobtain individual video feeds corresponding to an event. In some ofthese embodiments, the data management module 5212 may receive a requestfrom the video transformation module 5208 for a particular video feedcorresponding to an event. In response to the request, the datamanagement module 5212 may return the requested video feed to the videotransformation module 5208. The video feed may have been pushed to thevideo application by an external resource (e.g., multimedia platform),or may be requested (pulled) from the external resource in response tothe request.

With reference to FIG. 47, the machine learning model may include activelearning and active quality assurance on a live spatiotemporal machinelearning workflow 4700 in accordance with the various embodiments. Themachine learning workflow 4700 includes a machine learning (ML)algorithm 4702 that may produce live and automatic machine learning (ML)classification output 4704 (with minimum delay) as well as selectedevents for human quality assurance (QA) 4708 based on livespatiotemporal data 4710. In embodiments, the live spatiotemporalmachine learning workflow 4700 includes the data from the human questionand answer sessions that may then be fed back into a machine learning(ML) algorithm 4720 (which may be the same as the ML algorithm 4702),which may be rerun on the corresponding segments of data, to produce atime-delayed classification output 4724 with improved classificationaccuracy of neighboring events, where the time delay corresponds to theQA process.

In embodiments, the machine learning workflow 4700 includes data fromthe QA process 4708 being fed into ML training data 4722 to improve theML algorithm models for subsequent segments such as improving on the MLalgorithm 4702 and/or the ML algorithm 4702. Live spatiotemporal data4730 may be aligned with other imperfect sources of data related to asequence of spatial-temporal events. In embodiments, the alignmentacross imperfect sources of data related to a sequence ofspatial-temporal events may include alignment using novel generalizeddistance metrics for spatiotemporal sequences combining event durations,ordering of events, additions/deletions of events, a spatial distance ofevents, and the like.

In embodiments, the systems and methods disclosed herein may includemodeling and dynamically interacting with an n-dimensional point-cloud.By way of this example, each point may be represented as an n-spherewhose radius may be determined by letting each n-sphere grow until itcomes into contact with a neighboring n-sphere from a specified subsetof the given point-cloud. This method may be similar to a Voronoidiagram in that may allocate a single n-dimensional cell for every pointin the given cloud, with two distinct advantages. The first advantageincludes that the generative kernel of each cell may also be itscentroid. The second advantage includes continuously changing shifts inthe resulting model when points are relocated in a continuous fashion(e.g., as a function of time in an animation, or the like). Inembodiments, ten basketball players may be represented as ten nodes thatare divided into two subsets of five teammates. At any given moment,each player's cell may be included in a circle extending in radius untilit comes to be mutually tangent with an opponent's cell. By way of thisexample, players on the same team will have cells that overlap.

In one embodiment, the user interface 5084 includes a navigation userinterface that supports the use of swiping gestures and/or tappinggestures to navigate content in related time-aligned content channels(e.g., media segments, video segments/video clips, or on-demand videostreams). That is, the navigation user interface generates a visualdisplay for the user to use gestures to navigate content selections,such as navigating related content for an event such as a sportingevent.

As previously discussed, the time-alignment may correspond toframe-level synchronization using, for example, the previously describedtracking techniques, the spatial and temporal indexing techniques, andtime-alignment. Any of the previously described machine learningtechniques may be used to generate customized content.

In one embodiment, the swiping gestures and/or tapping gestures areinput into the navigation user interface by a user by physical orproximate contact with a physical touchscreen, such as by using theirfinger or a stylus. However, more generally similar gestures could beinput virtually (e.g., via virtual reality or augmented realityglasses). Moreover, the function of swiping and tapping may be emulated,in some cases, on some non-touchscreen user interfaces. The gestures aredetected and interpreted as commands to navigate content. In oneimplementation, video clips of an event are organized into storylinesand the user interface permits gesture-based navigation betweendifferent storylines and navigation within individual storylines.

FIG. 53 illustrates a system for generation of dynamic video andnavigation through a gesture-based user interface (UI) according to anexemplary and non-limiting embodiment. FIG. 53 is a variation of FIG.50, further including a gesture detection module 5305 to detect gestures(e.g., swiping up, swiping down, swiping left, swiping right, tapping).

The swiping up, down, left, and right are with respect to the verticaland horizontal directions associated with the display. Thus, a swipe ina vertical direction (“a vertical swipe”) includes detecting eitherswiping up or swiping down with respect to a vertical axis of thedisplay. A swipe in a horizontal direction (“a horizontal swipe”)includes either detecting swiping left or swiping right with respect toa horizontal axis of the display.

In some embodiments, the choice of swiping up or swiping down is anarbitrary design choice. For example, if a swipe down opens up adiscovery screen and a swipe up toggles between different video angles,it would be understood that these two options could be provided with theopposite ordering of a swipe up opening up a discover screen and a swipedown opening up a video with a different angle. Thus, in someembodiments it is more general to refer to a swipe in a first verticaldirection and a swipe in a second (opposite) vertical direction insteadof up/down. Similarly, in some embodiments it is more general to referto a swipe in a first horizontal direction and a swipe in a second(opposite) horizontal direction instead of left/right.

Other information for a gesture may also be detected and used innavigation, such as detecting a relative location on the user interfacewhere a gesture occurred (e.g., left, center, or right of display).Other attributes of the gestures could also, in theory, be detected,such as swiping speed, swiping length, etc. While detecting a single tapis one option, more generally multiple taps and/or a speed of tappingmay be detected. The navigation module 5310 supports using the detectedgestures as navigation commands. A relationship module 5315 may beprovided to store information on relationships between content, such asrelationships defining storylines and storyline elements.

In one implementation, the gesture detection module 5405 monitorsgestures (e.g., swipes and taps) entered in a touchscreen portion of auser interface 5084 of user device 5080. The navigation module 5310interprets the detected touchscreen gestures as navigation commands withrespect to a set of defined relationships between related content. Asexamples, the touchscreen may be used to request media segments (e.g.,video clips or text images) related in different ways to the currentmedia segment, to indicate when to move on to the next media segment,and in some implementations to pull up additional information such assports statistics, augmented replays, or different camera angles.

In some embodiments, the relationships define concatenated sequences inwhich a sequence of navigation commands leads from one media segment (orvideo stream) to another media segment. For example, the storylineelements in a storyline have an organization to define a sequence of theindividual elements with respect to each other (e.g., define whichstoryline elements come next). In one embodiment, a storyline organizesrelated content according to a theme or other plan, such as storylineorganizing a set of media segments (e.g., video clips or text) accordingto a theme. As an illustrative example, a storyline for a sporting eventmay include a set of storyline elements for an individual player or ateam. As another example, a storyline for a sporting event may supportdifferent camera angles for watching highlight clips, or differentaugmented clips. Each media segment in a storyline has a correspondingscreen in regards to the display of media.

In one embodiment, each storyline element screen displays dynamic videocontent (e.g., a media video clip or video feed), although moregenerally individual storyline element screens may also displaygraphics, animations, statistics, photos, text information, or otherinformation. A storyline may have an arbitrary number of storylineelements (e.g., 1, 2, . . . N storyline elements). In the general case,there may be a set, M, of different storylines (e.g., storyline 1,storyline 2, . . . storyline M).

FIG. 54 illustrates an example of an individual storyline havingstoryline elements according to an exemplary and non-limitingembodiment. A wireframe drawing is shown for the purposes ofillustration of the storyline of a navigation user interface. Thestoryline has a set of storyline elements capable of being navigated. Inthis example, the storyline for a basketball player has a set ofstoryline elements, where each story line element corresponds to ascreen element for display on a user device 5080. In this example,several of the storyline element screens have playable video content(e.g., highlight video clips) while the others have text, pictures, orgraphics. However, it will be understood that the number and content ofeach screen element in a storyline may be varied.

Many different variations on how the detected gestures are used fornavigation are contemplated. For example, one set of gestures may beused to navigate between storylines. Another set of gestures may be usedto navigate within storylines in order to navigate to differentstoryline elements and access any optional or augmented content. In oneembodiment horizontal swiping is used to navigate between storylines andother gestures (vertical swiping and/or tapping) are used to navigatewithin individual storylines.

FIG. 55 illustrates an example in which a gesture is used to skip toanother element within a storyline. In one embodiment, a tapping gestureis used to skip to the next element. In one implementation, adistinction is made between tapping towards the right edge of the screenversus tapping towards the left edge of the screen. In oneimplementation, tapping on the right jumps to the next storyline elementwithin a storyline and tapping on the left jumps to the previousstoryline element.

FIG. 56 illustrates an example of performing a vertical swiping gestureto display additional discovery information related to a particularstoryline element screen. For example, swiping down may open updiscovery information, which may for example include text, graphics, orvideo content.

FIG. 57 illustrates an example of performing a vertical swiping gestureto toggle modes or angles for storyline video clips. For example,swiping up may be used as a gesture to toggle modes, or angles forstoryline video clips. Alternatively, the vertical swiping gesture maybe used to access analytical views, insights, or other materials.

FIG. 58 illustrates an example of using a navigation user interface fornavigation between storylines. A wireframe drawing is shown for thepurposes of illustration. For example, a set of storylines may include astoryline A (e.g., game recap), a storyline B, (e.g., a particularplayer), a storyline C (e.g., a listing of top sports elements, such astop 10 dunks for basketball), and a storyline D (e.g., shows related tothe event). In this example, a horizontal swiping gesture is detectedand used to navigate to another storyline. In one embodiment, swipingright causes the storyline to jump to the next storyline on the right.Swiping left causes the storyline screen to jump to the previousstoryline on the left.

One aspect of the media system is that in one embodiment the livetracking and machine learning/artificial intelligence system producesmultiple frame-synchronized, augmented videos of sporting events, andgenerates highlight clips that are organized into storylines of relatedclips. Users can navigate related content with the user interface usinggestures, such as swiping gestures and taps, to perform navigationoperation such as cycling through the various camera angles andaugmented modes of the same highlight clip, i.e. the same sequence ofevents in the game. Since the different versions of the video clip areframe-synchronized, the switching between frames/modes occurs withoutinterrupting the time evolution of the events in the video clip. Thatis, the improved tracking and associated temporal and spatial alignmentfacilitates a mode of operation, in one embodiment, in which users cannavigate the content without experiencing a visual interruption of thetime evolution of the events in the video clip. This provides animproved user experience. The user can use gestures, such as tapping tojump to other video clips in the storyline without interrupting the timeevolution of the events.

As previously discussed, the user interface can be applied not only tohighlight video clips, but also to live and video-on-demand streams ofevents. For example, in one embodiment a swiping gesture is used toswitch between frame-synchronized streams of the same event fromdifferent camera angles, or different augmented versions of the videostreams. Alternatively, a tapping gesture could be used to perform theswitching. In one embodiment, for live or video-on-demand streams ofgames, the left and right navigation jumps the user to the previous ornext play in a game. In one implementation, the left navigation jumps tothe previous play in the game and the right navigation jumps to the nextplay in the game. For example, swiping left could result jumping theuser interface to the previous play in the game and swiping right couldjump the user interface to the next game in the play. That is, in thisexample the game is organized as a storyline whose elements are plays inthe game, e.g., segments of live play between dead balls.

In embodiments, the systems and methods disclosed herein may include amethod for modeling locale as a function of time, some other specifiedor predetermined variable, or the like. In embodiments, coordinates of agiven point or plurality of points are repeatedly sampled over a givenwindow of time. By way of this example, the sampled coordinates may thenbe used to generate a convex hull, and this procedure may be repeated asdesired and may yield a plurality of hulls that may be stacked for adiscretized view of spatial variability over time. In embodiments, asingle soccer player might have their location on a pitch sampled everysecond over the course of two minutes leading to a point cloud oflocation data and an associated convex hull. By way of this example, theprocess may begin anew with each two-minute window and the fullassemblage of generated hulls may be, for example, rendered in atranslucent fashion and may be layered so as to yield a map of the givenplayer's region of activity.

In embodiments, the systems and methods disclosed herein may include amethod for sampling and modeling data by applying the recursive logic ofa quadtree to a topologically deformed input or output space. Inembodiments, the location of shots in a basketball game may be sampledin arc-shaped bins, which may be partitioned by angle-of-incidence tothe basket and the natural logarithm of distance from the basket, and,in turn, yielding bins which may be subdivided and visualized accordingto the same rules governing a rectilinear quadtree.

In embodiments, the systems and methods disclosed herein may include amethod for modeling multivariate point-cloud data such that locationcoordinates map to the location, while velocity (or some other relevantvector) may be represented as a contour map of potential displacementsat various time intervals. In embodiments, a soccer player running downa pitch may be represented by a node surrounded by nested ellipses eachindicating a horizon of displacement for a given window of time.

In embodiments, the systems and methods disclosed herein may include amethod for modeling and dynamically interacting with a directed acyclicgraph such that every node may be rendered along a single line, whilethe edges connecting nodes may be rendered as curves deviating from thisline in accordance with a specified variable. In embodiments, theseedges may be visualized as parabolic curves wherein the height of eachmay correspond to the flow, duration, latency, or the like of theprocess represented by the given edge.

The methods and systems disclosed herein may include methods and systemsfor enabling a user to express preferences relating to display of videocontent and may include using machine learning to develop anunderstanding of at least one event, one metric related to the event, orrelationships between events, metrics, venue, or the like within atleast one video feed to determine at least one type for the event;automatically, under computer control, extracting the video contentdisplaying the event and associating the machine learning understandingof the type for the event with the video content in a video content datastructure; providing a user interface by which a user can indicate apreference for at least one type of content; and upon receiving anindication of the preference by the user, retrieving at least one videocontent data structure that was determined by the machine learning tohave content of the type preferred by the user and providing the userwith a video feed containing the content of the preferred type.

In embodiments, the user interface is of at least one of a mobileapplication, a browser, a desktop application, a remote control device,a tablet, a touch screen device, a virtual reality or augmented realityheadset, and a smart phone. In embodiments, the user interface furthercomprises an element for allowing a user to indicate a preference as tohow content will be presented to the user. In embodiments, the machinelearning further comprises determining an understanding of a context forthe event and the context is stored with the video content datastructure. In embodiments, the user interface further comprises anelement for allowing a user to indicate a preference for at least onecontext. In embodiments, upon receiving an indication of a preferencefor a context, video content corresponding to the context preference isretrieved and displayed to the user. In embodiments, the contextcomprises at least one of the presence of a preferred player in thevideo feed, a preferred matchup of players in the video feed, apreferred team in the video feed, and a preferred matchup of teams inthe video feed. In embodiments, the user interface allows a user toselect at least one of a metric and a graphic element to be displayed onthe video feed, wherein at least one of the metric and the graphic isbased at least in part on the machine understanding.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interacting with videocontent method and may include taking a video feed; using machinelearning to develop an understanding of an event within the video feed,the understanding including identifying context information relating tothe event; automatically, under computer control, extracting the contentdisplaying the event and associating the extracted content with thecontext information; producing a video content data structure thatincludes the associated context information; and using the contextinformation for a plurality of such video content data structures togenerate, automatically under computer control, producing a story orvideo clip that includes the video content data structure, wherein thecontent of the story is based on a user preference. In embodiments, theuser preference for a type of content is based on at least one of a userexpressed preference and a preference that is inferred based on userinteraction with an item of content.

The methods and systems disclosed herein may include methods and systemsfor enabling a user to express preferences relating to display of videocontent and may include a machine learning facility for developing anunderstanding of at least one event within at least one video feed todetermine at least one type for the event; a video production facilityfor automatically, under computer control, extracting the video contentdisplaying the event and associating the machine learning understandingof the type for the event with the video content in a video content datastructure; a server for serving data to a user interface by which a usercan indicate a preference for at least one type of content; and uponreceiving at the server an indication of the preference by the user,retrieving at least one video content data structure that was determinedby the machine learning to have content of the type preferred by theuser and providing the user with a video feed containing the content ofthe preferred type.

In embodiments, the user interface is of at least one of a mobileapplication, a browser, a desktop application, a remote control device,a tablet, and a smart phone. In embodiments, the user interface furthercomprises an element for allowing a user to indicate a preference as tohow content will be presented to the user. In embodiments, the machinelearning further comprises determining an understanding of a context forthe event and the context is stored with the video content datastructure. In embodiments, the user interface further comprises anelement for allowing a user to indicate a preference for at least onecontext. In embodiments, upon receiving an indication of a preferencefor a context, video content corresponding to the context preference isretrieved and displayed to the user. In embodiments, the contextcomprises at least one of the presence of a preferred player in thevideo feed, a preferred matchup of players in the video feed, apreferred team in the video feed, and a preferred matchup of teams inthe video feed. In embodiments, the user interface allows a user toselect at least one of a metric and a graphic element to be displayed onthe video feed, wherein the metric is based at least in part on themachine understanding.

The methods and systems disclosed herein may include methods and systemsdelivering personalized video content and may include using machinelearning to develop an understanding of at least one event within atleast one video feed to determine at least one type for the event;automatically, under computer control, extracting the video contentdisplaying the event and associating the machine learning understandingof the type for the event with the video content in a video content datastructure; developing a personal profile for a user based on at leastone of expressed preferences of the user, information about the user,and information collected about actions taken by the user with respectto at least one type of video content; and upon receiving an indicationof the user profile, retrieving at least one video content datastructure that was determined by the machine learning to have content ofthe type likely to be preferred by the user based on the user profile.

The methods and systems disclosed herein may include methods and systemsfor delivering personalized video content and may include using machinelearning to develop an understanding of at least one event within atleast one video feed to determine at least one type for the event,wherein the video feed is a video feed for a professional game; usingmachine learning to develop an understanding of at least one eventwithin a data feed relating to the motion of a non-professional player;based on the machine learning understanding of the video feed for theprofessional game and the data feed of the motion of thenon-professional player, automatically, under computer control,providing an enhanced video feed that represents the non-professionalplayer playing within the context of the professional game. Inembodiments, the methods and systems may further include providing afacility having cameras for capturing 3D motion data and capturing videoof a non-professional player to provide the data feed for thenon-professional player. In embodiments, the non-professional player isrepresented by mixing video of the non-professional player with video ofthe professional game. In embodiments, the non-professional player isrepresented as an animation having attributes based on the data feedabout the non-professional player.

The methods and systems disclosed herein may also include one or more ofthe following features and capabilities: spatiotemporal patternrecognition (including active learning of complex patterns and learningof actions such as P&R, postups, play calls); hybrid methods forproducing high quality labels, combining automated candidate generationfrom XYZ data, and manual refinement; indexing of video by automatedrecognition of game clock; presentation of aligned optical and video;new markings using combined display, both manual and automated (via posedetection etc.); metrics: shot quality, rebounding, defense and thelike; visualizations such as Voronoi, heatmap distribution, etc.;embodiment on various devices; video enhancement with metrics &visualizations; interactive display using both animations and video;gesture and touch interactions for sports coaching and commentatordisplays; and cleaning of XYZ data using, for example, HMM, PBP, video,hybrid validation.

Further details as to data cleaning 204 are provided herein. Raw inputXYZ is frequently noisy, missing, or wrong. XYZ data is also deliveredwith attached basic events such as possession, pass, dribble, shot.These are frequently incorrect. This is important because eventidentification further down the process (Spatiotemporal PatternRecognition) sometimes depends on the correctness of these basic events.As noted above, for example, if two players' XY positions are switched,then “over” vs. “under” defense would be incorrectly switched, since theplayers' relative positioning is used as a critical feature for theclassification. Also, PBP data sources are occasionally incorrect.First, one may use validation algorithms to detect all events, includingthe basic events such as possession, pass, dribble, shot, and reboundthat are provided with the XYZ data. Possession/Non-possession may use aHidden Markov Model to best fit the data to these states. Shots andrebounds may use the possession model outputs, combined with 1)projected destination of the ball, and 2) PBP information. Dribbles maybe identified using a trained ML algorithm and also using the output ofthe possession model.

Specifically, once possessions are determined, dribbles may beidentified with a hidden Markov model. The hidden Markov model consistsof three states:

1. Holding the ball while the player is still able to dribble.

2. Dribbling the ball.

3. Holding the ball after the player has already dribbled.

A player starts in State 1 when he gains possession of the ball. At alltimes players are allowed to transition to either their current state,or the state with a number one higher than their current state, if sucha state exists.

The players' likelihood of staying in their current state ortransitioning to another state may be determined by the transitionprobabilities of the model as well as the observations. The transitionprobabilities may be learned empirically from the training data. Theobservations of the model consist of the player's speed, which is placedinto two categories, one for fast movement, and one for slow movement,as well as the ball's height, which is placed into categories for lowand high height. The cross product of these two observations representsthe observation space for the model. Similar to the transitionprobabilities, the observation probabilities, given a particular state,may be learned empirically from the training data. Once theseprobabilities are known, the model is fully characterized and may beused to classify when the player is dribbling on unknown data.

Once it is known that the player is dribbling, it remains to bedetermined when the actual dribbles occur. This may be done with aSupport Vector Machine that uses domain specific information about theball and player, such as the height of the ball as a feature todetermine whether at that instant the player is dribbling. A filteringpass may also be applied to the resulting dribbles to ensure that theyare sensibly separated, so that for instance, two dribbles do not occurwithin 0.04 seconds of each other.

Returning to the discussion of the algorithms, these algorithms decreasethe basic event labeling error rate by a significant factor, such asabout 50%. Second, the system has a library of anomaly detectionalgorithms to identify potential problems in the data. These includetemporal discontinuities (intervals of missing data are flagged);spatial discontinuities (objects traveling is a non-smooth motion,“jumping”); interpolation detection (data that is too smooth, indicatingthat post-processing was done by the data supplier to interpolatebetween known data points in order to fill in missing data). Thisproblem data is flagged for human review so that events detected duringthese periods are subject to further scrutiny.

Spatio-player tracking may be undertaken in at least two types, as wellas in a hybrid combined type. For tracking with broadcast video, thebroadcast video is obtained from multiple broadcast video feeds.Typically, this will include a standard “from the stands view” from thecenter stands midway-up, a backboard view, a stands view from a lowerangle from each corner, and potentially other views. Optionally, PTZ(pan tilt zoom) sensor information from each camera is also returned. Analternative is a Special Camera Setup method. Instead of broadcastfeeds, this uses feeds from cameras that are mounted specifically forthe purposes of player tracking. The cameras are typically fixed interms of their location, pan, tilt, zoom. These cameras are typicallymounted at high overhead angles; in the current instantiation, typicallyalong the overhead catwalks above the court. A Hybrid/Combined Systemmay be used. This system would use both broadcast feeds and feeds fromthe purpose-mounted cameras. By combining both input systems, accuracyis improved. Also, the outputs are ready to be passed on to the DataFXpipeline for immediate processing, since the DataFX will be paintinggraphics on top of the already-processed broadcast feeds. Wherebroadcast video is used, the camera pose must be solved in each frame,since the PTZ may change from frame to frame. Optionally, cameras thathave PTZ sensors may return this info to the system, and the PTZ inputsare used as initial solutions for the camera pose solver. If thisinitialization is deemed correct by the algorithm, it will be used asthe final result; otherwise, refinement will occur until the systemreceives a useable solution. As described above, players may beidentified by patches of color on the court. The corresponding positionsare known since the camera pose is known, and we can perform the properprojections between 3D space and pixel space.

Where purpose mounted cameras are used, multiple levels of resolutionmay be involved. Certain areas of the court or field require moresensitivity, e.g., on some courts, the color of the “paint” area makesit difficult to track players when they are in the paint. Extra cameraswith higher dynamic range and higher zoom are focused on these areas.The extra sensitivity enables the computer vision techniques to trainseparate algorithms for different portions of the court, tuning eachalgorithm to its type of inputs and the difficulty of that task.

In a combination system, by combining the fixed and broadcast videofeeds, the outputs of a player tracking system can feed directly intothe DataFX production, enabling near-real-time DataFX. Broadcast videomay also produce high-definition samples that can be used to increaseaccuracy.

The methods and systems disclosed herein may include methods and systemsfor enabling interaction with a broadcast video content stream and mayinclude a machine learning facility for developing an understanding ofat least one event within a video feed for a video broadcast, theunderstanding including identifying context information relating to theevent; and a touch screen user interface by which a broadcaster caninteract with the video feed, wherein the options for broadcasterinteraction are based on the context information, wherein theinteraction with the touch screen controls the content of the broadcastvideo event. In embodiments, the touch screen interface is a largescreen adapted to be seen by viewers of the video broadcast as thebroadcaster uses the touch screen. In embodiments, a smaller touchscreen is used by a commentator on air to control the informationcontent being displayed, and the images/video on the touch screen issimultaneously displayed on a larger screen that is filmed and broadcastor is simultaneously displayed directly in the broadcast feed. Inembodiments, the broadcaster can select from a plurality ofcontext-relevant metrics, graphics, or combinations thereof to bedisplayed on the screen. In embodiments, the broadcaster can display aplurality of video feeds that have similar contexts as determined by themachine learning facility. In embodiments, the similarity of contexts isdetermined by comparing events within the video feeds. In embodiments,the broadcaster can display a superimposed view of at least two videofeeds to facilitate a comparison of events from a plurality of videofeeds. In embodiments, the comparison is of similar players fromdifferent, similar, or identical time periods. In embodiments, asimilarity of players is determined by machine understanding of thecharacteristics of the players from the different time periods. Inembodiments, the broadcaster can display a plurality of highlights thatare automatically determined by a machine understanding of a live sportsevent that is the subject of the video feed. In embodiments, thehighlights are determined based on similarity to highlights that havebeen identified for other events.

The methods and systems disclosed herein may include methods and systemsfor enabling interaction with a broadcast video content stream and mayinclude developing a machine learning understanding of at least oneevent within a video feed for a video broadcast, the understandingincluding identifying context information relating to the event; andproviding a touch screen user interface by which a broadcaster caninteract with the video feed, wherein the options for broadcasterinteraction are based on the context information, wherein theinteraction with the touch screen controls the content of the broadcastvideo event. In embodiments, the touch screen interface is a largescreen adapted to be seen by viewers of the video broadcast as thebroadcaster uses the touch screen. In embodiments, the broadcaster canselect from a plurality of context-relevant metrics to be displayed onthe screen. In embodiments, the broadcaster can display a plurality ofvideo feeds that have similar contexts as determined by the machinelearning facility. In embodiments, the similarity of contexts isdetermined by comparing events within the video feeds. In embodiments,the broadcaster can display a superimposed view of at least two videofeeds to facilitate a comparison of events from a plurality of videofeeds. In embodiments, the comparison is of similar players fromdifferent time periods.

In embodiments, a similarity of players is determined by machineunderstanding of the characteristics of the players from the differenttime periods. In embodiments, the broadcaster can display a plurality ofhighlights that are automatically determined by a machine understandingof a live sports event that is the subject of the video feed. Inembodiments, the highlights are determined based on similarity tohighlights that have been identified for other events.

The methods and systems disclosed herein may include methods and systemsfor enabling interaction with a broadcast video content stream and mayinclude taking a video feed; using machine learning to develop anunderstanding of an event within the video feed, the understandingincluding identifying context information relating to the event;automatically, under computer control, extracting the content displayingthe event and associating the extracted content with the contextinformation; producing a video content data structure that includes theassociated context information; and providing an application by which auser can interact with the video content data structure, wherein theoptions for user interaction are based on the context information,wherein the interaction with the video content data structure controlsthe presentation of a broadcast video event on a display screen.

Methods and systems disclosed herein may include tracklet stitching.Optical player tracking results in short to medium length tracklets,which typically end when the system loses track of a player or theplayer collides (or passes close to) with another player. Using teamidentification and other attributes, algorithms can stitch thesetracklets together.

Where a human being is in the loop, systems may be designed for rapidinteraction and for disambiguation and error handling. Such a system isdesigned to optimize human interaction with the system. Novel interfacesmay be provided to specify the motion of multiple moving actorssimultaneously, without having to match up movements frame by frame.

In embodiments, custom clipping is used for content creation, such asinvolving OCR. Machine vision techniques may be used to automaticallylocate the “score bug” and determine the location of the game clock,score, and quarter information. This information is read and recognizedby OCR algorithms. Post-processing algorithms using various filteringtechniques are used to resolve issues in the OCR. Kalman filtering/HMMsmay be used to detect errors and correct them. Probabilistic outputs(which measure the degree of confidence) assist in this errordetection/correction.

Sometimes, a score is nonexistent or cannot be detected automatically(e.g., sometimes during PIP or split screens). In these cases, remaininginconsistencies or missing data is resolved with the assistance of humaninput. Human input is designed to be sparse so that labelers do not haveto provide input at every frame. Interpolation and other heuristics areused to fill in the gaps. Consistency checking is done to verify gameclock.

For alignment 2112, as discussed in connection with FIG. 21, anotheradvance is to use machine vision techniques to verify some of theevents. For example, video of a made shot will typically show the scorebeing increased or will show a ball going through a hoop. Either kind ofautomatic observation serves to help the alignment process result in thecorrect video frames being shown to the end user.

In accordance with an exemplary and non-limiting embodiment, augmentedor enhanced video with extracted semantics-based experience is providedbased, at least in part, on 3D position/motion data. In accordance withother exemplary embodiments, there is provided embeddable app contentfor augmented video with an extracted semantics-based experience. In yetanother exemplary embodiment, there is provided the ability toautomatically detect the court/field, and relative positioning of thecamera, in (near) real time using computer vision techniques. This maybe combined with automatic rotoscoping of the players in order toproduce dynamic augmented video content.

The methods and systems disclosed herein may include methods and systemsfor embedding video content in an application and may include taking avideo feed; using machine learning to develop an understanding of anevent within the video feed, the understanding including identifyingcontext information relating to the event; automatically, under computercontrol, extracting the content displaying the event and associating theextracted content with the context information; producing a videocontent data structure that includes the associated context information;taking an application that displays video content; and embedding thevideo content data structure in the application. In embodiments, theuser interface of the application offers the user the option to controlthe presentation of the video content from the video content datastructure in the application. In embodiments, the control of thepresentation is based on at least one of a user preference and a userprofile. In embodiments, the application is a mobile application thatprovides a story about an event and wherein the video content datastructure comprises at least one of a content card and a digital stillimage.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application that allows user interaction withvideo content and may include a video ingestion facility for taking avideo feed; a machine learning facility for developing an understandingof an event within the video feed, the understanding includingidentifying context information relating to the event; and a videoproduction facility for automatically, under computer control,extracting the content displaying the event, associating the extractedcontent with the context information and producing a video content datastructure that includes the associated context information; and usingthe context information for a plurality of such video content datastructures to generate, automatically under computer control, a storythat includes a sequence of the video content data structures. Inembodiments, the content of the story is based on a user profile that isbased on at least one of an expressed user preference, information abouta user interaction with video content, and demographic information aboutthe user. In embodiments, the methods and systems may further includedetermining a pattern relating to a plurality of events in the videofeed and associating the determined pattern with the video content datastructure as additional context information. In embodiments, the patternrelates to a highlight event within the video feed. In embodiments, thehighlight event is associated with at least one of a player and a team.In embodiments, the embedded application allows a user to indicate atleast one of a player and a team for which the user wishes to obtainvideo feeds containing the highlight events. In embodiments, the patternrelates to a comparison of events occurring at least one of within thevideo feed or within a plurality of video feeds. In embodiments, thecomparison is between events occurring over time. In embodiments, theembedded application allows a user to select at least one player toobtain a video providing a comparison between the player and at leastone of a past representation of the same player and a representation ofanother player. In embodiments, the pattern is a cause-and-effectpattern related to the occurrence of a following type of event after theoccurrence of a pre-cursor type of event. In embodiments, the embeddedapplication allows the user to review video cuts in a sequence thatdemonstrate the cause-and-effect pattern. In embodiments, theapplication provides a user interface for allowing a user to enter atleast one of text and audio input to provide a narrative for a sequenceof events within the video feed. In embodiments, the user may select asequence of video events from within the feed for display in theapplication. In embodiments, upon accepting the user narrative, thesystem automatically generates an electronic story containing the eventsfrom the video feed and the narrative.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application that allows user interaction withvideo content and may include taking a video feed; using a machinelearning facility to develop an understanding of an event within thevideo feed, the understanding including identifying context informationrelating to the event; and automatically, under computer control,extracting the content displaying the event, associating the extractedcontent with the context information and producing a video content datastructure that includes the associated context information. Inembodiments, the methods and systems may further include using thecontext information for a plurality of such video content datastructures to generate, automatically under computer control, a storythat includes a sequence of the video content data structures.

In embodiments, the user may interact with an application, such as on aphone, laptop, or desktop, or with a remote control, to control thedisplay of broadcast video. As noted above in connection withinteraction with a mobile application, options for user interaction maybe customized based on the context of an event, such as by offeringoptions to display context-relevant metrics for the event. Theseselections may be used to control the display of broadcast video by theuser, such as by selecting preferred, context-relevant metrics thatappear as overlays, sidebars, scrolling information, or the like on thevideo display as various types of events take place in the video stream.For example, a user may select settings for a context like a three pointshot attempt, so that when the video displays three point shot attempts,particular metrics (e.g., the average success percentage of the shooter)are shown as overlays above the head of the shooter in the video.

The methods and systems disclosed herein may include methods and systemsfor personalizing content for each type of user based on determining thecontext of the content through machine analysis of the content and basedon an indication by the user of a preference for a type of presentationof the content.

The methods and systems disclosed herein may include methods and systemsfor enabling a user to express preferences relating to display of videocontent and may include: taking a video feed; using machine learning todevelop an understanding of an event within the video feed, theunderstanding including identifying context information relating to theevent; automatically, under computer control, extracting the contentdisplaying the event and associating the extracted content with thecontext information; producing a video content data structure thatincludes the associated context information; and providing a userinterface by which a user can indicate a preference for how content thatis associated with a particular type of context will be presented to theuser.

In embodiments, a user may be presented with an interface element for amobile application, browser, desktop application, remote control,tablet, smart phone, or the like, for indicating a preference as to howcontent will be presented to the user. In embodiments, the preferencemay be indicated for a particular context, such a context determined bya machine understanding of an event. In embodiments, a user may selectto see certain metrics, graphics or additional information overlaid ontop of the existing broadcast for certain types of semantic events suchas players expected field goal percentage when they possess the ball orthe type and effectiveness of defense being played on a pick and roll.

The methods and systems disclosed herein may include methods and systemsfor automatically generating stories/content based on the personalprofile of a viewer and their preferences or selections ofcontextualized content.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interacting with videocontent method and may include taking a video feed; using machinelearning to develop an understanding of an event within the video feed,the understanding including identifying context information relating tothe event; automatically, under computer control, extracting the contentdisplaying the event and associating the extracted content with thecontext information; producing a video content data structure thatincludes the associated context information; and using the contextinformation for a plurality of such video content data structures togenerate, automatically under computer control, a story that includesthe video content data structures, wherein the content of the story isbased on a user preference. In embodiments, the user preference for atype of content is based on at least one of a user expressed preferenceand a preference that is inferred based on user interaction with an itemof content.

In embodiments, items of content that are associated, based on machineunderstanding, with particular events in particular contexts can belinked together, or linked with other content, to produce modifiedcontent such as stories. For example, a game summary, such as extractedfrom an online report about an event, may be augmented withmachine-extracted highlight cuts that correspond to elements featured inthe game summary, such as highlights of important plays, images ofparticular players, and the like. These stories can be customized for auser, such as linking a story about a game played by the user's favoriteteam with video cuts of the user's favorite player that were takenduring the game.

The methods and systems disclosed herein may include methods and systemsfor using machine learning to extract context information andsemantically relevant events and situations from a video content stream,such that the events and situations may be presented according to thecontext of the content.

The methods and systems disclosed herein may include methods and systemsfor embedding video content in an application and may include taking avideo feed; using machine learning to develop an understanding of anevent within the video feed, the understanding including identifyingcontext information relating to the event; automatically, under computercontrol, extracting the content displaying the event and associating theextracted content with the context information; producing a videocontent data structure that includes the associated context information;taking an application that displays video content; and embedding thevideo content data structure in the application, wherein the location ofthe embedded video content in the application is based on the contextinformation.

In embodiments, context-identified video cuts can be used to enrich orenhance applications, such as by embedding the cuts in relevantlocations in the applications. For example, a mobile applicationdisplaying entertainment content may be automatically populated withvideo cuts of events that are machine-extracted and determined to be ofthe appropriate type (based on context), for the application. A videogame application can be enhanced, such as by including real video cutsof plays that fit a particular context (e.g., showing a pick-and-rollplay where players A and B are matched up against players C and D in areal game, and the same matchup is determined to occur in the videogame).

To facilitate embedding the application, a set of protocols, such asAPIs, may be defined, by which available categories (such as semanticcategories, types of contexts, types of events, and the like) arespecified, such that an application may call for particular types ofevents, which can, in turn, be embedded in the application. Similarly,an application may be constructed with appropriate pointers, calls,objects, or the like, that allow a designer to specify, and call for,particular types of events, which may be automatically extracted from alibrary of machine-extracted, context-identified events and thenembedded where appropriate into the application code.

In embodiments, an application may provide stories about events, such assporting events, and the machine-extracted content may include contentcards or digital stills that are tagged by context so that they can beplaced in appropriate locations in a story. The application can provideautomatically generated content and stories, enhanced by content from alive game. In embodiments, an application may recommend video clipsbased on the use of keywords that match machine learned semantics thatenable users to post or share video clips automatically tailored to textthat they are writing. For example, clips may be recommended thatinclude the presence of a particular player, that include a particulartype of play (e.g., “dunks”) and/or that are from a particular timeperiod (e.g., “last night,” etc.). In accordance with an exemplary andnon-limiting embodiment, there is described a method for the extractionof events and situations corresponding to semantically relevantconcepts. In yet other embodiments, semantic events may be translatedand cataloged into data and patterns.

The methods and systems disclosed herein may include methods and systemsfor embedding content cards or digital stills with contextualizedcontent stories/visualizations into a mobile application. They mayinclude automatically generated content, such as stories, extracted froma live game delivered to users via an application, such as a mobileapplication, an augmented reality glasses application, a virtual realityglasses application, or the like. In embodiments, the application is amobile application that provides a story about an event and wherein thevideo content data structure comprises at least one of a content cardand a digital still image.

The methods and systems disclosed herein may include methods and systemsfor applying contextualized content from actual sporting events to videogames to improve the reality of the game play.

The methods and systems disclosed herein may include methods and systemsfor improving a video game and may include taking a video feed; usingmachine learning to develop an understanding of at least one first realevent within the video feed, the understanding including identifyingcontext information relating to the first real event; taking a gameevent coded for display within a video game; matching the contextinformation for the real event with the context of the game event in thevideo game; comparing the display of the game event to the video for thereal event; and modifying the coding of the game event based on thecomparison.

In embodiments, context information can be used to identify video cutsthat can be used to improve video games, such as by matching the contextof a real event with a similar context in a coded video game event,comparing the video for the real event with the video game display of asimilar event, and modifying the video event to provide a more faithfulsimulation of the real event. The methods and systems disclosed hereinmay include methods and systems for taking the characteristics of a usereither from a video capture of their recreational play or through usergenerated features and importing the user's avatar into a video game.The methods and systems disclosed herein may include methods and systemsfor interactive contextualized content that can be filtered and adjustedvia a touch screen interface. In embodiments, the user interface is atouch screen interface.

The methods and systems disclosed herein may include methods and systemsfor real time display of relevant fantasy and betting metrics overlaidon a live game feed. The methods and systems disclosed herein mayinclude methods and systems for real time adjustment of betting linesand/or additional betting option creation based on in-game contextualcontent.

The methods and systems disclosed herein may include methods and systemsfor taking a video feed and using machine learning to develop anunderstanding of at least one first event within the video feed. Theunderstanding includes identifying context information relating to thefirst event. The methods and systems also include determining a metricbased on the machine understanding. The metric is relevant to at leastone of a wager and a fantasy sports outcome. The methods and systemsinclude presenting the metric as an overlay for an enhanced video feed.

In embodiments, the metrics described throughout this disclosure may beplaced as overlays on video feeds. For example, metrics calculated basedon machine-extracted events that are relevant to betting lines, fantasysports outcomes, or the like, can be presented as overlays, scrollingelements, or the like on a video feed. The metrics to be presented canbe selected based on context information, such as showing fantasymetrics for players who are on screen at the time or showing the bettingline where a scoring play impacts the outcome of a bet. As noted above,the displays may be customized and personalized for a user, such asbased on that user's fantasy team for a given week or that user's wagersfor the week.

The methods and systems disclosed herein may include methods and systemsfor taking a video feed of a recreational event; using machine learningto develop an understanding of at least one event within the video feed,the understanding including identifying context information relating tothe event; and based on the machine understanding, providing contentincluding information about a player in the recreational event based onthe machine understanding and the context. The methods and systems mayfurther include providing a comparison of the player to at least oneprofessional player according to at least one metric that is based onthe machine understanding.

In embodiments, machine understanding can be applied to recreationalvenues, such as for capturing video feeds of recreational games,practices, and the like. Based on machine understanding, highlightclips, metrics, and the like, as disclosed throughout this disclosure,may be extracted by processing the video feeds, including machineunderstanding of the context of various events within the video. Inembodiments, metrics, video, and the like can be used to provide playerswith personalized content, such as a highlight reel of good plays, or acomparison to one or more professional players (in video cuts, or withsemantically relevant metrics). Context information can allowidentification of similar contexts between recreational and professionalevents, so that a player can see how a professional acted in a contextthat is similar to one faced by the recreational player. The methods andsystems may enable the ability to use metrics and events recorded from avideo stream to enable the creation of a recreational fantasy sportsgame with which users can interact. The methods and systems may enablethe ability for to recognize specific events or metrics from arecreational game and compare them to similar or parallel events from aprofessional game to help coach a recreational player or team or for thecreation of a highlight reel that features both recreational andprofessional video cuts.

The methods and systems disclosed herein may include methods and systemsfor providing enhanced video content and may include using machinelearning to develop an understanding of a plurality of events within atleast one video feed to determine at least one type for each of theplurality of events; extracting a plurality of video cuts from the videofeed and indexing the plurality of video cuts based on at least one typeof event determined by the understanding developed by machine learning;and making the indexed and extracted video cuts available to a user. Inembodiments, the user is enabled to at least one of edit, cut, and mixthe video cuts to provide an enhanced video containing at least one ofthe video cuts. In embodiments, the user is enabled to share theenhanced video. In embodiments, the methods and systems may furtherinclude indexing at least one shared, enhanced video with the semanticunderstanding of the type of events in that was determined by machinelearning. In embodiments, the methods and systems may further includeusing the index information for the shared, enhanced video to determinea similarity between the shared, enhanced video and at least one othervideo content item. In embodiments, the similarity is used to identifyadditional extracted, indexed video cuts that may be of interest to theuser. In embodiments, the similarity is used to identify other users whohave shared similarly enhanced video. In embodiments, the similarity isused to identify other users who are likely to have an interest in theshared, enhanced video. In embodiments, the methods and systems mayfurther include recommending at least one of the shared, enhanced videoand one of the video cuts based on an understanding of the preferencesof the other users. In embodiments, the similarity is based at least inpart on user profile information for users who have indicated aninterest in the video cut and the other video content item.

The methods and systems disclosed herein may include methods and systemsfor providing enhanced video content and may include using machinelearning to develop an understanding of a plurality of events within atleast one video feed to determine at least one type for each of theplurality of events; extracting a plurality of video cuts from the videofeed and indexing the plurality of video cuts to form an indexed set ofextracted video cuts, wherein the indexing is based on at least one typeof event determined by the understanding developed by machine learning;determining at least one pattern relating to a plurality of events inthe video feed; adding the determined pattern information to the indexfor the indexed set of video cuts; and making the indexed and extractedvideo cuts available to a user. In embodiments, the user is enabled toat least one of edit, cut, and mix the video cuts to provide an enhancedvideo containing at least one of the video cuts. In embodiments, theuser is enabled to share the enhanced video. In embodiments, the videocuts are clustered based on the patterns that exist within the videocuts. In embodiments, the pattern is determined automatically usingmachine learning and based on the machine understanding of the events inthe video feed. In embodiments, the pattern is a highlight event withinthe video feed. In embodiments, the highlight event is presented to theuser when the indexed and extracted video cut is made available to theuser. In embodiments, the user is prompted to watch a longer video feedupon viewing the indexed and extracted video cut.

In accordance with an exemplary and non-limiting embodiment, there isprovided a touch screen or other gesture-based interface experiencebased, at least in part, on extracted semantic events.

The methods and systems disclosed herein may include methods and systemsfor machine extracting semantically relevant events from 3Dmotion/position data captured at a venue, calculating a plurality ofmetrics relating to the events, and presenting the metrics in a videostream based on the context of the video stream.

The methods and systems disclosed herein may include methods and systemsfor producing machine-enhanced video streams and may include taking avideo feed from 3D motion and position data from a venue; using machinelearning to develop an understanding of at least one first event withinthe video feed, the understanding including identifying contextinformation relating to the first event; calculating a plurality ofmetrics relating to the events; and producing an enhanced video streamthat presents the metrics in the video stream, wherein the presentationof at least one metric is based on the context information for the eventwith which the metric is associated in the video stream.

In embodiments, semantically relevant events determined by machineunderstanding of 3D motion/position data for an event from a venue canbe used to calculate various metrics, which may be displayed in thevideo stream of the event. Context information, which may be determinedbased on the types and sequences of events, can be used to determinewhat metrics should be displayed at a given position within the videostream. These metrics may also be used to create new options for usersto place wagers on or be integrated into a fantasy sports environment.

The methods and systems disclosed herein may include methods and systemsenabling a user to cut or edit video based on machine learned contextand share the video clips. These may further include allowing a user tointeract with the video data structure to produce an edited video datastream that includes the video data structure. In embodiments, theinteraction includes at least one of editing, cutting, and sharing avideo clip that includes the video data structure. The methods andsystems may enable the ability for users to interact with video cutsthrough an interface to enhance the content with graphics or metricsbased on a pre-set set of options, and then share a custom cut andenhanced clip. The methods and systems may include the ability toautomatically find similarity in different video clips based on semanticcontext contained in the clips, and then cluster clips together or torecommend additional clips for viewing. The methods and systems mayinclude the ability to extract contextualized content from a feed of arecreational event to immediately deliver content to players, includingcomparing a recreational player to a professional player based onmachine learned understanding of player types.

In accordance with an exemplary and non-limiting embodiment, there isdescribed a second screen interface unique to extracted semantic eventsand user selected augmentations. In yet other embodiments, the secondscreen may display real-time, or near real time, contextualized content.

In accordance with further exemplary and non-limiting embodiments, themethods and systems disclosed herein may include methods and systems fortaking a video feed; using machine learning to develop an understandingof an event within the video feed, the understanding includingidentifying context information relating to the event; automatically,under computer control, extracting the content displaying the event andassociating the extracted content with the context information; andproducing a video content data structure that includes the associatedcontext information. In embodiments, the methods and systems may furtherinclude determining a plurality of semantic categories for the contextinformation and filtering a plurality of such video content datastructures based on the semantic categories. In embodiments, the methodsand systems may further include matching the events that occur in onevideo feed to those that occur in a separate video feed such that thesemantic understanding captured in the first video feed can be used toat least one of filter and cut a separate second video feed based on thesame events. In embodiments, the methods and systems may further includedetermining a pattern relating to a plurality of the events andproviding a content data structure based on the pattern. In embodiments,the pattern comprises a plurality of important plays in a sports eventthat are identified based on comparison to similar plays from previoussports events. In embodiments, the pattern comprises a plurality ofplays in a sports event that is determined to be unusual based oncomparison to video feeds from other sports events. In embodiments, themethods and systems may further include extracting semantic events overtime to draw a comparison of at least one of a player and a team overtime.

In embodiments, the methods and systems may further includesuperimposing video of events extracted from video feeds from at leasttwo different time periods to illustrate the comparison. In embodiments,the methods and systems may further include allowing a user to interactwith the video data structure to produce an edited video data streamthat includes the video data structure. In embodiments, the interactionincludes at least one of editing, mixing, cutting, and sharing a videoclip that includes the video data structure. In embodiments, the methodsand systems may further include enabling users to interact with thevideo cuts through a user interface to enhance the video content with atleast one graphic element selected from a menu of options.

In embodiments, the methods and systems may further include enabling auser to share the enhanced video content. In embodiments, the methodsand systems may further include enabling a user to find similar videoclips based on the semantic context identified in the clips. Inembodiments, the methods and systems may further include using the videodata structure and the context information to construct modified videocontent for a second screen that includes the video data structure. Inembodiments, the content for the second screen correlates to the timingof an event displayed on a first screen. In embodiments, the content forthe second screen includes a metric determined based on the machineunderstanding, wherein the metric is selected based on the contextinformation.

The methods and systems disclosed herein may include methods and systemsfor displaying contextualized content of a live event on a second screenthat correlates to the timing of the live event on the first screen.These may include using the video data structure and the contextinformation to construct modified video content for a second screen thatincludes the video data structure. In embodiments, the content for thesecond screen correlates to the timing of an event displayed on a firstscreen. In embodiments, the content for the second screen includes ametric determined based on the machine understanding, wherein the metricis selected based on the context information.

In embodiments, machine extracted metrics and video cuts can bedisplayed on a second screen, such as a tablet, smart phone, or smartremote control screen, such as showing metrics that are relevant to whatis happening, in context, on a main screen.

The methods and systems disclosed herein may include methods and systemsfor an ingestion facility adapted or configured to ingest a plurality ofvideo feeds; a machine learning system adapted or configured to applymachine learning on a series of events in a plurality of video feeds inorder to develop an understanding of an event within the video feed, theunderstanding including identifying context information relating to theevent; an extraction facility adapted or configured to automatically,under computer control, extract the content displaying the event andassociate the extracted content with the context information; and avideo publishing facility for producing a video content data structurethat includes the associated context information. In embodiments, themethods and systems may further include an analytic facility adapted orconfigured to determine a plurality of semantic categories for thecontext information and filter a plurality of such video content datastructures based on the semantic categories. In embodiments, the methodsand systems may further include a matching engine adapted or configuredto match the events that occur in one video feed to those that occur ina separate video feed such that the semantic understanding captured inthe first video feed can be used to at least one of filter and cut aseparate second video feed based on the same events. In embodiments, themethods and systems may further include a pattern recognition facilityadapted or configured to determine a pattern relating to a plurality ofthe events and providing a content data structure based on the pattern.

The methods and systems disclosed herein may include methods and systemsfor displaying machine extracted, real time, contextualized contentbased on machine identification of a type of event occurring in a livevideo stream.

The methods and systems disclosed herein may include methods and systemsfor taking a video feed; using machine learning to develop anunderstanding of an event within the video feed, the understandingincluding identifying context information relating to the event;automatically, under computer control, extracting the content displayingthe event and associating the extracted content with the contextinformation; and producing a video content data structure that includesthe associated context information. The methods and systems disclosedherein may include methods and systems for providing context informationin video cuts that are generated based on machine extracted cuts thatare filtered by semantic categories.

The methods and systems disclosed herein may include methods and systemsfor determining a plurality of semantic categories for the contextinformation and filtering a plurality of the video content datastructures based on the semantic categories. The methods and systemsdisclosed herein may include methods and systems for matching the eventsthat occur in one video feed to those that occur in a separate videofeed such that the semantic understanding captured in the first videofeed can be used to filter and cut a separate second video feed based onthese same events.

The methods and systems disclosed herein may include methods and systemsfor enabling user interaction with a mobile application that displaysextracted content, where the user interaction is modified based on thecontext of the content (e.g., the menu is determined by context).

The methods and systems disclosed herein may include methods and systemsfor enabling an application allowing user interaction with video contentand may include an ingestion facility adapted or configured to access atleast one video feed, wherein the ingestion facility may be executing onat least one processor; a machine learning facility operating on the atleast one video feed to develop an understanding of an event within thevideo feed, the understanding including identifying context informationrelating to the event; an extraction facility adapted or configured toautomatically, under computer control, extract the content displayingthe event and associate the extracted content with the contextinformation; a video production facility adapted or configured toproduce a video content data structure that includes the associatedcontext information; and an application having a user interface by whicha user can interact with the video content data structure, wherein theoptions for user interaction are based on the context information. Inembodiments, the application is a mobile application. In embodiments,the application is at least one of a smart television application, avirtual reality headset application and an augmented realityapplication. In embodiments, the user interface is a touch screeninterface. In embodiments, the user interface allows a user to enhancethe video feed by selecting a content element to be added to the videofeed. In embodiments, the content element is at least one of a metricand a graphic element that is based on the machine understanding. Inembodiments, the user interface allows the user to select content for aparticular player of a sports event. In embodiments, the user interfaceallows the user to select content relating to a context involving thematchup of two particular players in a sports event.

In embodiments, the system takes at least two video feeds from differenttime periods, the machine learning facility determines a context theincludes a similarity between at least one of a plurality of players anda plurality of plays in the two feeds and the user interface allows theuser to select at least one of the players and the plays to obtain avideo feed that illustrates a comparison. In embodiments, the userinterface includes options for at least one of editing, cutting, andsharing a video clip that includes the video data structure.

In embodiments, the video feed comprises 3D motion camera data capturedfrom a live sports venue. In embodiments, the ability of the machinelearning facility to develop the understanding is developed by feedingthe machine learning facility a plurality of events for which contexthas already been identified.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interaction with videocontent and may include taking at least one video feed; applying machinelearning on the at least one video feed to develop an understanding ofan event within the video feed, the understanding including identifyingcontext information relating to the event; automatically, under computercontrol, extracting the content displaying the event and associating theextracted content with the context information; producing a videocontent data structure that includes the associated context information;and providing a mobile application having a user interface by which auser can interact with the video content data structure, wherein theoptions for user interaction are based on the context information. Inembodiments, the user interface is a touch screen interface. Inembodiments, the user interface allows a user to enhance the video feedby selecting a content element to be added to the video feed. Inembodiments, the content element is at least one of a metric and agraphic element that is based on the machine understanding. Inembodiments, the user interface allows the user to select content for aparticular player of a sports event. In embodiments, the user interfaceallows the user to select content relating to a context involving thematchup of two particular players in a sports event.

In embodiments, the system takes at least two video feeds from differenttime periods, the machine learning facility determines a context theincludes a similarity between at least one of a plurality of players anda plurality of plays in the two feeds and the user interface allows theuser to select at least one of the players and the plays to obtain avideo feed that illustrates a comparison. In embodiments, the userinterface includes options for at least one of editing, cutting, andsharing a video clip that includes the video data structure. Inembodiments, the video feed comprises 3D motion camera data capturedfrom a live sports venue. In embodiments, the ability of the machinelearning facility to develop the understanding is developed by feedingthe machine learning facility a plurality of events for which contexthas already been identified.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interacting with videocontent and may include taking a video feed; using machine learning todevelop an understanding of an event within the video feed, theunderstanding including identifying context information relating to theevent; automatically, under computer control, extracting the contentdisplaying the event and associating the extracted content with thecontext information; producing a video content data structure thatincludes the associated context information; and providing a mobileapplication by which a user can interact with the video content datastructure, wherein the options for user interaction are based on thecontext information.

In embodiments, machine extracted content, with associated contextinformation, may be provided to users via a mobile application, throughwhich the users may display and interact with the content, such as byselecting particular types of content based on a desired semanticcategory (such as by selecting the category in list, menu, or the like),playing content (including pausing, rewinding, fast forwarding, and thelike), and manipulating content (such as positioning content within adisplay window, zooming, panning, and the like). In embodiments, thenature of the permitted interaction may be governed by the contextinformation associated with the content, where the context informationis based on a machine understanding of the content and its associatedcontext. For example, where the content is related to a particular typeof play within a context of an event like a game, such as reboundingopportunities in basketball, the user may be permitted to select from aset of metrics that are relevant to rebounding, so that the selectedmetrics from a context-relevant set are displayed on the screen with thecontent. If the context is different, such as if the content relates toa series of pick-and-roll plays by a particular player, differentmetrics may be made available for selection by the user, such asstatistics for that player, or metrics appropriate for pick-and-rolls.Thus, the machine-extracted understanding of an event, including contextinformation, can be used to customize the content displayed to the user,including to allow the user to select context-relevant information fordisplay.

The methods and systems disclosed herein may include methods and systemsfor allowing a user to control a presentation of a broadcast videoevent, where the options for control are based on a context of thecontent as determined by machine extraction of semantically relevantevents from the content.

In accordance with an exemplary and non-limiting embodiment, there isdescribed a method for “painting” translated semantic data onto aninterface.

In accordance with an exemplary and non-limiting embodiment, there isdescribed spatiotemporal pattern recognition based, at least in part, onoptical XYZ alignment for semantic events. In yet other embodiments,there is described the verification and refinement of spatiotemporalsemantic pattern recognition based, at least in part, on hybridvalidation from multiple sources.

In accordance with an exemplary and non-limiting embodiment, there isdescribed human identified video alignment labels and markings forsemantic events. In yet other embodiments, there is described machinelearning algorithms for spatiotemporal pattern recognition based, atleast in part, on human identified video alignment labels for semanticevents.

In accordance with an exemplary and non-limiting embodiment, there isdescribed automatic game clock indexing of video from sporting eventsusing machine vision techniques, and cross-referencing this index with asemantic layer that indexes game events. The product is the ability toquery for highly detailed events and return the corresponding video innear real-time.

In accordance with an exemplary and non-limiting embodiment, there isdescribed unique metrics based, at least in part, on spatiotemporalpatterns including, for example, shot quality, rebound ratings(positioning, attack, conversion) and the like.

In accordance with an exemplary and non-limiting embodiment, there isdescribed player tracking using broadcast video feeds.

In accordance with an exemplary and non-limiting embodiment, there isdescribed player tracking using a multi-camera system.

In accordance with an exemplary and non-limiting embodiment, there isdescribed video cut-up based on extracted semantics. A video cut-up is aremix made up of small clips of video that are related to each other insome meaningful way. The semantic layer enables real-time discovery anddelivery of custom cut-ups. The semantic layer may be produced in one oftwo ways: (1) Video combined with data produces a semantic layer, or (2)video directly to a semantic layer. Extraction may be through ML orhuman tagging. In some exemplary embodiments, video cut-up may be based,at least in part, on extracted semantics, controlled by users in astadium and displayed on a Jumbotron. In other embodiments, video cut-upmay be based, at least in part, on extracted semantics, controlled byusers at home and displayed on broadcast TV. In yet other embodiments,video cut-up may be based, at least in part, on extracted semantics,controlled by individual users and displayed on the web, tablet, ormobile for that user. In yet other embodiments, video cut-up may bebased, at least in part, on extracted semantics, created by anindividual user, and shared with others. Sharing could be throughinter-tablet/inter-device communication, or via mobile sharing sites.

In accordance with further exemplary and non-limiting embodiments, themethods and systems disclosed herein may include methods and systems forenabling an application allowing user interaction with video content andmay include an ingestion facility for taking at least one video feed; amachine learning facility operating on the at least one video feed todevelop an understanding of an event within the video feed, theunderstanding including identifying context information relating to theevent; an extraction facility for automatically, under computer control,extracting the content displaying the event and associating theextracted content with the context information; a video productionfacility for producing a video content data structure that includes theassociated context information; and an application having a userinterface by which a user can interact with the video content datastructure, wherein the options for user interaction are based on thecontext information. In embodiments, the application is a mobileapplication. In embodiments, the application is at least one of a smarttelevision application, a virtual reality headset application and anaugmented reality application. In embodiments, the user interface is atouch screen interface. In embodiments, the user interface allows a userto enhance the video feed by selecting a content element to be added tothe video feed. In embodiments, the content element is at least one of ametric and a graphic element that is based on the machine understanding.In embodiments, the user interface allows the user to select content fora particular player of a sports event. In embodiments, the userinterface allows the user to select content relating to a contextinvolving the matchup of two particular players in a sports event.

In embodiments, the system takes at least two video feeds from differenttime periods, the machine learning facility determines a context theincludes a similarity between at least one of a plurality of players anda plurality of plays in the two feeds and the user interface allows theuser to select at least one of the players and the plays to obtain avideo feed that illustrates a comparison. In embodiments, the userinterface includes options for at least one of editing, cutting, andsharing a video clip that includes the video data structure.

In embodiments, the video feed comprises 3D motion camera data capturedfrom a live sports venue. In embodiments, the ability of the machinelearning facility to develop the understanding is developed by feedingthe machine learning facility a plurality of events for which contexthas already been identified.

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interaction with videocontent and may include taking at least one video feed; applying machinelearning on the at least one video feed to develop an understanding ofan event within the video feed, the understanding including identifyingcontext information relating to the event; automatically, under computercontrol, extracting the content displaying the event and associating theextracted content with the context information; producing a videocontent data structure that includes the associated context information;and providing a mobile application having a user interface by which auser can interact with the video content data structure, wherein theoptions for user interaction are based on the context information. Inembodiments, the user interface is a touch screen interface. Inembodiments, the user interface allows a user to enhance the video feedby selecting a content element to be added to the video feed. Inembodiments, the content element is at least one of a metric and agraphic element that is based on the machine understanding. Inembodiments, the user interface allows the user to select content for aparticular player of a sports event. In embodiments, the user interfaceallows the user to select content relating to a context involving thematchup of two particular players in a sports event.

In embodiments, the system takes at least two video feeds from differenttime periods, the machine learning facility determines a context theincludes a similarity between at least one of a plurality of players anda plurality of plays in the two feeds and the user interface allows theuser to select at least one of the players and the plays to obtain avideo feed that illustrates a comparison. In embodiments, the userinterface includes options for at least one of editing, cutting, andsharing a video clip that includes the video data structure.

In embodiments, the video feed comprises 3D motion camera data capturedfrom a live sports venue. In embodiments, the ability of the machinelearning facility to develop the understanding is developed by feedingthe machine learning facility a plurality of events for which contexthas already been identified.

The methods and systems disclosed herein may include methods and systemsfor an analytic system and may include a video ingestion facility foringesting at least one video feed; a machine learning facility thatdevelops an understanding of at least one event within the video feed,wherein the understanding identifies at least a type of the event and atime of the event in an event data structure; a computing architectureenabling a model that takes one or more event data structures as inputand applies at least one calculation to transform the one or more eventdata structures into an output data structure; and a data transportlayer of the computing architecture for populating the model with theevent data structures as input to the model. In embodiments, the outputdata structure includes at least one prediction. In embodiments, theprediction is of an outcome of at least one of a sporting event and atleast one second event occurring within a sporting event. Inembodiments, the video feed is of a live sporting event, wherein theprediction is made during the live sporting event, and wherein theprediction relates to the same sporting event. In embodiments, theprediction is based on event data structures from a plurality of videofeeds.

In embodiments, the prediction is used for at least one of placing awager, setting a line for a wager, interacting with a fantasy program,setting a parameter of a fantasy program, providing insight to a coachand providing information to a fan. In embodiments, the model takesinputs from a plurality of data sources in addition to the event datastructures obtained from the video feed. In embodiments, the methods andsystems may further include a pattern analysis facility that takes aplurality of the event data structures and enables analysis of patternsamong the event data structures. In embodiments, the pattern analysisfacility includes at least one tool selected from the group consistingof a pattern visualization tool, a statistical analysis tool, a machinelearning tool, and a simulation tool. In embodiments, the methods andsystems may further include a second machine learning facility forrefining the model based on outcomes of a plurality of predictions madeusing the model.

The methods and systems disclosed herein may include methods and systemsfor an analytic method and may include ingesting at least one video feedin a computing platform capable of handling video data; developing anunderstanding of at least one event within the video feed using machinelearning, wherein the understanding identifies at least a type of theevent and a time of the event in an event data structure; providing acomputing architecture that enables a model that takes one or more eventdata structures as input and applies at least one calculation totransform the one or more event data structures into an output datastructure; and populating the model with the event data structures asinput to the model. In embodiments, the output data structure includesat least one prediction. In embodiments, the prediction is of an outcomeof at least one of a sporting event and at least one-second eventoccurring within a sporting event. In embodiments, the video feed is ofa live sporting event, wherein the prediction is made during the livesporting event, and wherein the prediction relates to the same sportingevent. In embodiments, the prediction is based on event data structuresfrom a plurality of video feeds. In embodiments, the prediction is usedfor at least one of placing a wager, setting a line for a wager,interacting with a fantasy program, setting a parameter of a fantasyprogram, providing insight to a coach and providing information to afan. In embodiments, the model takes inputs from a plurality of datasources in addition to the event data structures obtained from the videofeed. In embodiments, the methods and systems may further includeproviding a pattern analysis facility that takes a plurality of theevent data structures and enables analysis of patterns among the eventdata structures. In embodiments, the pattern analysis facility includesat least one tool selected from the group consisting of a patternvisualization tool, a statistical analysis tool, a machine learningtool, and a simulation tool. In embodiments, the methods and systems mayfurther include at least one of providing and using a second machinelearning facility to refine the model based on outcomes of a pluralityof predictions made using the model.

The methods and systems disclosed herein may include methods and systemsfor taking a video feed; using machine learning to develop anunderstanding of a semantically relevant event within the video feed;indexing video segments of the video feed with information indicatingthe semantically relevant events identified within the feed by themachine learning; and applying machine learning to a plurality of thesemantically relevant events to determine a pattern of events. Inembodiments, the pattern is within a video feed. In embodiments, thepattern is across a plurality of video feeds. In embodiments, thepattern corresponds to a narrative structure. In embodiments, thenarrative structure corresponds to a recurring pattern of events. Inembodiments, the narrative structure relates to a sporting event andwherein the pattern relates to at least one of a blow-out victorypattern, a comeback win pattern, a near comeback pattern, aback-and-forth game pattern, an individual achievement pattern, aninjury pattern, a turning point moment pattern, a close game pattern,and a team achievement pattern.

In embodiments, the indexed video segments are arranged to support thenarrative structure. In embodiments, the arranged segments are providedin an interface for developing a story using the segments that followthe narrative structure and wherein a user may at least one of edit andenter additional content for the story. In embodiments, summary contentfor the narrative structure is automatically generated, under computercontrol, to provide a story that includes the video sequences. Inembodiments, the methods and systems may further include delivering aplurality of the automatically generated stories at least one of from adefined time period and of a defined type, allowing a user to indicatewhether they like or dislike the delivered stories, and using theindications to inform later delivery of at least one additional story.In embodiments, the pattern is relevant to a prediction. In embodiments,the prediction is related to a wager, and the pattern corresponds tosimilar patterns that were used to make predictions that resulted insuccessful wagers in other situations.

The methods and systems disclosed herein may include methods and systemsfor machine-extracting semantically relevant events from a video contentstream and determining a pattern relating to the events. The methods andsystems also include providing a content stream based on the pattern. Inembodiments, the content stream is used to provide coaching informationbased on the pattern. In embodiments, the content stream is used toassist prediction of an outcome in a fantasy sports contest. Inembodiments, the pattern is used to provide content for a viewer of asporting event.

The methods and systems disclosed herein may include methods and systemsfor machine-extracting semantically relevant events from a video contentstream; determining a pattern relating to the events; storing thepattern information with the extracted events; and providing a user withthe option to view and interact with the patterns, wherein at least oneof the patterns and the interaction options are personalized based on aprofile of the user. In embodiments, the profile is based on at leastone of user indication of a preference, information about actions of theuser, and demographic information about the user. In embodiments, thepattern comprises at least one of a trend and a statistic that iscurated to correspond with the user profile. In embodiments, the patternrelates to a comparison of a professional athlete to another athlete. Inembodiments, the other athlete is the user and the comparison is basedon a playing style of the user as determined by at least one ofinformation indicated by the user and a video feed of the user. Inembodiments, the pattern relates to an occurrence of an injury. Inembodiments, the pattern information is used to provide coaching toprevent an injury. In embodiments, the methods and systems may furtherinclude automatically generating, under computer control, an injuryprevention regimen based on the pattern and based on information aboutthe user.

The methods and systems disclosed herein may include methods and systemsfor machine-extracting semantically relevant events from a video contentstream, determining a pattern relating to the events, and providing acontent stream based on the pattern. The methods and systems may furtherinclude determining a pattern relating to a plurality of the events andproviding a content data structure based on the pattern.

In embodiments, machine-extracted information about events and contextsmay be used to determine one or more patterns, such as by analyzing timeseries, correlations, and the like in the machine-extracted events andcontexts. For example, tendencies of a team to follow running a certainplay with a particular play may be determined by comparing instances ofthe two plays over time. Embodiments may include extracting particularlyinteresting or potential “game changing” plays by understanding thecontext of an individual event and comparing it to similar events fromprevious games. Embodiments may include extracting situations or playsthat are particularly rare or unique by understanding the context of anindividual event and comparing it to similar events from previous games.Embodiments may include extracting semantic events over time to draw acomparison of a player's or team's trajectory over time andsuperimposing video to draw out this comparison.

The methods and systems disclosed herein may include methods and systemsfor a model to predict the outcome of a game or events within a gamebased on a contextualized understanding of a live event for use inbetting/fantasy, coaching, augmented fan experiences, or the like.

The methods and systems disclosed herein may include methods and systemsfor an analytic system and may include taking a video feed; usingmachine learning to develop an understanding of at least one first eventwithin the video feed, the understanding including identifying contextinformation relating to the first event; taking a model used to predictthe outcome of at least one of a live game and at least one second eventwithin a live game; and populating the model with the machineunderstanding of the first event and the context information to producea prediction of an outcome of at least one of the game and the secondevent. In embodiments, the model is used for at least one of placing awager, setting a line for a wager, interacting with a fantasy program,setting a parameter of a fantasy program, providing insight to a coachand providing information to a fan.

In embodiments, machine-extracted event and context information can beused to populate one or more predictive models, such as models used forbetting, fantasy sports, coaching, and entertainment. The machineunderstanding, including various metrics described throughout thisdisclosure, can provide or augment other factors that are used topredict an outcome. For example, outcomes from particular matchups canbe machine extracted and used to predict outcomes from similar matchupsin the future. For example, based on the machine understood context of amoment in an individual game, and the machine understanding of similarmoments from previous games, a model can be created to predict theoutcome of an individual play or a series of plays on which anindividual can place a bet or on which a betting line may be set.

In embodiments, the methods and systems disclosed herein may includemethods and systems for suggestions of bets to make based on patterns ofpreviously successful bets. For example, a user may be prompted with anoption to place a bet based on previous betting history on similarevents or because a particular moment is an opportunistic time to placea bet based on the context of a game and other user generatedpreferences or risk tolerances.

The methods and systems disclosed herein may include methods and systemsfor automated storytelling, such as the ability to use patternsextracted from semantic events, metrics derived from tracking data, andcombinations thereof to populate interesting stories about the content.

The methods and systems disclosed herein may include methods and systemsfor enabling automated generation of stories and may include taking avideo feed; using machine learning to develop an understanding of asemantically relevant event within the video feed, the understandingincluding identifying context information relating to the event;providing a narrative structure for a story, wherein the narrativestructure is arranged based on the presence of semantic types of eventsand the context of those events; and automatically, under computercontrol, generating a story following the narrative structure, whereinthe story is populated based on a sequence of the machine-understoodevents and the context information.

In embodiments, patterns from semantic events may be used to populatestories. Various narrative structures can be developed, corresponding tocommon patterns of events (e.g., stories about blow-out victories,comeback wins, back-and-forth games, games that turned on big moments,or the like). Machine extracting of events and contexts can allowidentification of patterns in the events and contexts that allowmatching to one or more of the narrative structures, as well aspopulation of the story with content for the events, such as video cutsor short written summaries that are determined by the machine extraction(e.g., “in the first quarter, Team A took the lead, scoring five timeson the pick-and-roll.”).

The methods and systems disclosed herein may include methods and systemsfor enabling a mobile application allowing user interacting with videocontent and may include taking a video feed; using machine learning todevelop an understanding of an event within the video feed, theunderstanding including identifying context information relating to theevent; automatically, under computer control, extracting the contentdisplaying the event and associating the extracted content with thecontext information; producing a video content data structure thatincludes the associated context information; and providing a mobileapplication by which a user can interact with the video content datastructure, wherein the options for user interaction are based on thecontext information.

In embodiments, machine extracted content, with associated contextinformation, may be provided to users via a mobile application, throughwhich the users may display and interact with the content, such as byselecting particular types of content based on a desired semanticcategory (such as by selecting the category in list, menu, or the like),playing content (including pausing, rewinding, fast forwarding, and thelike), and manipulating content (such as positioning content within adisplay window, zooming, panning, and the like). In embodiments, thenature of the permitted interaction may be governed by the contextinformation associated with the content, where the context informationis based on a machine understanding of the content and its associatedcontext. For example, where the content is related to a particular typeof play within a context of an event like a game, such as reboundingopportunities in basketball, the user may be permitted to select from aset of metrics that are relevant to rebounding, so that the selectedmetrics from a context-relevant set are displayed on the screen with thecontent. If the context is different, such as if the content relates toa series of pick-and-roll plays by a particular player, differentmetrics may be made available for selection by the user, such asstatistics for that player, or metrics appropriate for pick-and-rolls.Thus, the machine-extracted understanding of an event, including contextinformation, can be used to customize the content displayed to the user,including to allow the user to select context-relevant information fordisplay.

The methods and systems disclosed herein may include methods and systemsfor allowing a user to control the presentation of a broadcast videoevent, where the options for control are based on a context of thecontent as determined by machine extraction of semantically relevantevents from the content.

In accordance with an exemplary and non-limiting embodiment, X, Y, and Zdata may be collected for purposes of inferring player actions that havea vertical component.

The methods and systems disclosed herein may employ a variety ofcomputer vision, machine learning, and/or active learning techniques andtools to extract, analyze and process data elements originating fromsources, such as, but not limited to, input data sources relating tosporting events and items in them, such as players, venues, items usedin sports (such as balls, pucks, and equipment), and the like. Thesedata elements may be available as video feeds in an example, such thatthe video feeds may be captured by image recognition devices, videorecognition devices, image and video capture devices, audio recognitiondevices, and the like, including by use of various devices andcomponents such as a camera (such as a tracking camera or broadcastcamera), a microphone, an image sensor, or the like. Audio feeds may becaptured by microphones and similar devices, such as integrated on orwith cameras or associated with independent audio capture systems. Inputfeeds may also include tracking data from chips or sensors (such aswearable tracking devices using accelerometers and other motionsensors), as well as data feeds about an event, such as a play-by-playdata feed, a game clock data feed, and the like. In the case of inputfeeds, facial recognition systems may be used to capture facial imagesof players, such as to assist in recognition of players (such as incases where player numbers are absent or obscured) and to capture andprocess expressions of players, such as emotional expressions,micro-expressions, or the like. These expressions may be associated withevents, such as to assist in machine understanding (e.g., an expressionmay convey that the event was exciting, meaningful, the like, that itwas disappointing to one constituency, that it was not important, or thelike). Machine understanding may thus be trained to recognizeexpressions and provide an expression-based understanding of events,such as to augment one or more data structures associated with an eventfor further use in the various embodiments described herein. Forexample, a video feed may be processed based on a machine understandingof expressions to extract cuts that made players of one team happy. Asanother example, a cut showing an emotional reaction (such as by aplayer, fan, teammate, or coach) to an event may be associated with acut of the event itself, providing a combined cut that shows the eventand the reaction it caused. The various embodiments described throughoutthis disclosure the involve machine understanding, extraction of cuts,creation of data structures that are used or processed for variouspurposes, combining cuts, augmenting data feeds, producing stories,personalizing content, and the like should all be understood toencompass, where appropriate, use of machine understanding of emotionalexpression within a video feed, including based on use of computervision techniques, including facial recognition techniques andexpression recognition techniques.

The computer vision, machine learning and/or active learning tools andtechniques (together referred to as computer-controlled intelligentsystems for simplicity herein) may receive the data elements fromvarious input feeds and devices as a set of inputs either in real-time(such as in case of a live feed or broadcast) or at a different time(such as in case of a delayed broadcast of the sporting or any otherevent) without limitations. The computer-controlled intelligent systemsmay process the set of inputs, apply machine learning and naturallanguage processing using artificial intelligence (AI) and naturallanguage processing (NLP) capabilities to produce a set of services andoutputs.

In an example, the set of services and outputs may signifyspatial-temporal positions of the players and sports accessories/objectssuch as a bat, ball, football, and the like. In an example, the set ofservices and outputs may represent spatial-temporal alignments of theinputs such as the video feeds, etc. For example, a broadcast video feedmay be aligned in time with another input feed, such as input from oneor more motion tracking cameras, inputs from player tracking systems(such as wearable devices), and the like. The set of services andoutputs may include machine understood contextual outputs involvingmachine learning or understanding that may be built using various levelsof artificial intelligence, algorithmic processes, computer-controlledtasks, custom rules, and the like, such as described throughout thisdisclosure. The machine understanding may include various levels ofsemantic identification, as well as information of position and speedinformation for various items or elements, identification of basicevents such as various types of shots and screens during a sportingevent, and identification of complex events or a sequence of events suchas various types of plays, higher level metrics and patterns involvingsuch as game trajectory, style of play, strengths and weaknesses ofteams and team members/players from each team, and the like. The machinelearning tools and input feed alignment may allow automatic generationof content and information such as statistics, predictions, comparisons,and analysis. The machine learning tools may further allow to generateoutputs based on a user query input such as to determine variouspredictive analytics for a particular team player in view of historicalshots and screens in a particular context, determine possibilities ofsuccess and failures in particular zones and game scenarios conditionedto particular user inputs, and the like. The machine understanding toolsmay simulate entire aspects of real-life sporting events on a computerscreen utilizing visualization and modeling examples. The services andoutputs generated by the intelligent computer-controlled systems may beused in a variety of ways such as generation of a live feed or a delayedfeed during a sporting event in real time or at a later broadcastingtime after the sporting event. The services and outputs may allowgenerating various analyses of statistics, trends, and strategy beforeevents or across multiple events. The services and outputs mayfacilitate an interactive user session to extract contextual detailsrelating to instantaneous sporting sessions of the sporting events inassociation with user defined queries, constraints, and rules.

In an example, the services and outputs generated by thecomputer-controlled intelligent systems may enable spatiotemporalanalysis of various game attributes and elements for exploring,learning, analyzing such sporting events and utilize analytics resultsto generate predictive models and predictive analytics for gamingstrategy. These services and outputs may provide valuable insights andlearnings that are otherwise not visible.

The methods and systems disclosed herein may employ delay-dependentcomputer vision and machine learning systems (or the intelligentcomputer-controlled systems) for providing delay-dependent services andoutputs with respect to the occurrence of a sporting event. The servicesand outputs as discussed herein may be employed in differentapplications with varying time delays relative to the actual occurrenceof the sporting event. For example, the actual event may occur at a timeT1 and the content feeding or broadcasting may occur at a time T2 with atime delay of T2-T1. The time delay may be small such as of a fewseconds so as the content is useful in a live commentary or augmentationof a live video. In such cases, the machine learning tools may forexample utilize real-time services and outputs and benefit from thespatiotemporal features and attributes to generate game patterns andautomatic validations during the event itself such as to highlightcertain event aspects in the commentary and/or validate momentarysessions when there are confusions during the event for decision making.The time delay may be longer in certain situations such as for replays,post-event analysis, predictive modeling, and future strategies, and thelike.

The methods and systems disclosed herein may support the provisioning ofthe services and outputs at various time delays by determiningprocessing steps and their order of execution according to delayrequirements. The system may be configured to operate such that theservices and outputs may be obtained at arbitrary times with anincreasing accuracy or time resolution or such that the system targetsspecific delay requirements as specified by users or defined inaccordance with intended applications. For example, if in anapplication, computational resources are insufficient to process allframes originating from input devices such as cameras etc. at maximumaccuracy at a video frame rate within a desired delay, then instead ofprocessing the input video frames in sequential orders, processing maybe ordered in such a way that at any time there is a uniform orapproximately uniform distribution of processed frames. In some cases,processing decisions may also be influenced by other computationalefficiency considerations for certain tasks that operate on videosegments, such as an opportunity to reuse certain computations acrosssuccessive frames in tracking algorithms. In some examples, processingtechniques such as inference and interpolation over processed frames maybe used to provide a tracking output whose accuracy and time resolutionimproves with delay as more frames are processed. If a target delay isspecified, each component of processing application (such as backgroundsubtraction, detection of various elements) may be assigned an executiontime budget within which to compute its output, such that the specifieddelay is met by a combination of the components. In some examples, thespecified time delays may also consider video qualities needed atsending destinations so as to ensure that enough computation resourcesare allocated for appropriate resolutions and transmission rates at thedestinations during broadcasting of the content. In certain cases, anormal resolution may be sufficient while in other cases a higherresolution may be needed. In various embodiments, the intelligentcomputer-controlled systems may be capable of defining appropriateresolutions, data transmission rates, and computation resourcesallocation in view of the delay requirements.

The methods and systems disclosed herein may facilitate enablingcalibration of a moving camera or any other image recognition device viatracking of moving points in a sporting event. Existing techniques forfinding unknown camera calibration parameters from captured images orvideos of sporting events rely on identifying a set of known locations,such as intersections of lines on the court or field. In accordance withsuch techniques, calibrating the moving camera as it changes itsposition or zooms across frames is challenging since there may be only afew of such known locations in the frames. The methods and systemsdisclosed herein may enable finding the calibration parameters of themoving or operator-controlled camera by using positions of moving pointslocated by an associated tracking system. In an example, these positionsmay represent locations and spatial coordinates of a player's or areferee's head or hand or legs in the sporting event which may beidentified by the tracking system. The tracking system may be an opticaltracking system or a chip-based tracking system, which may be configuredto determine positions of locations tags. In various examples, severalother types of camera control, calibration, and position determiningsystems may be employed along with the tracking systems. For example, afixed spotting camera may be used to capture a view and a moving cameracontained within the tracking system may be used to capture thepositions of the moving points in the frames. The moving camera may beconfigured to perform several functions such as zoom, tilt, pan, and thelike. The tracking system may be configured to perform calibration andidentification of the positions based on a tracking algorithm that mayexecute pre-defined instructions to compute relevant informationnecessary to drive the tracking system across the frames.

The methods and systems disclosed herein may facilitate enablingpre-processing of images from calibrated cameras to improve objectdetection and recognition. The methods and systems disclosed herein mayenable providing for accurate detection and recognition of humans, suchas players or referees, and objects, such as a ball, a game clock,jersey numbers and the like with better performance and lowercomplexity. In embodiments, the tasks of object detection andrecognition may be performed on the basis of knowledge of knowncalibration parameters of the cameras in the tracking system and knownproperties of the objects being detected such as their size,orientation, or positions etc. For example, perspectives and distortionsintroduced by the cameras can be undone by applying a transformationsuch that the objects being detected may have a consistent scale andorientation in transformed images. The transformed images may be used asinputs to detection and recognition algorithms by image processingdevices so as to enable faster and more accurate object detection andrecognition performance with lower complexity as compared to performingobject detection and recognition directly on original images. In suchcases, an output generated by the image processing devices may be usedas inputs, along with other inputs described herein, to enable or refinethe various machine learning and algorithmic capabilities describedthroughout this disclosure. In some embodiments, machine learningcapabilities may be introduced to build improved processing utilizingmachine learning tools as discussed above in the document.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The processor may be part of aserver, client, network infrastructure, mobile computing platform,stationary computing platform, or other computing platforms. A processormay be any kind of computational or processing device capable ofexecuting program instructions, codes, binary instructions, and thelike. The processor may be or include a signal processor, digitalprocessor, embedded processor, microprocessor, or any variant such as aco-processor (math co-processor, graphic co-processor, communicationco-processor and the like) and the like that may directly or indirectlyfacilitate execution of program code or program instructions storedthereon. In addition, the processor may enable execution of multipleprograms, threads, and codes. The threads may be executed simultaneouslyto enhance the performance of the processor and to facilitatesimultaneous operations of the application. By way of implementation,methods, program codes, program instructions and the like describedherein may be implemented in one or more thread. The thread may spawnother threads that may have assigned priorities associated with them;the processor may execute these threads based on priority or any otherorder based on instructions provided in the program code. The processormay include memory that stores methods, codes, instructions, andprograms as described herein and elsewhere. The processor may access astorage medium through an interface that may store methods, codes, andinstructions as described herein and elsewhere. The storage mediumassociated with the processor for storing methods, programs, codes,program instructions or other type of instructions capable of beingexecuted by the computing or processing device may include but may notbe limited to one or more of a CD-ROM, DVD, memory, hard disk, flashdrive, RAM, ROM, cache, and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,client, firewall, gateway, hub, router, or other such computer and/ornetworking hardware. The software program may be associated with aserver that may include a file server, print server, domain server,Internet server, intranet server and other variants such as secondaryserver, host server, distributed server, and the like. The server mayinclude one or more of memories, processors, computer readable media,storage media, ports (physical and virtual), communication devices, andinterfaces capable of accessing other servers, clients, machines, anddevices through a wired or a wireless medium, and the like. The methods,programs or codes as described herein and elsewhere may be executed bythe server. In addition, other devices required for execution of methodsas described in this application may be considered as a part of theinfrastructure associated with the server.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers, andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope. Inaddition, any of the devices attached to the server through an interfacemay include at least one storage medium capable of storing methods,programs, code and/or instructions. A central repository may provideprogram instructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, Internet client, intranetclient and other variants such as secondary client, host client,distributed client, and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for execution of methods as describedin this application may be considered as a part of the infrastructureassociated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers, andthe like. Additionally, this coupling and/or connection may facilitateremote execution of program across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more location without deviating from the scope. Inaddition, any of the devices attached to the client through an interfacemay include at least one storage medium capable of storing methods,programs, applications, code and/or instructions. A central repositorymay provide program instructions to be executed on different devices. Inthis implementation, the remote repository may act as a storage mediumfor program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM, and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements.

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network having multiplecells. The cellular network may either be frequency division multipleaccess (FDMA) network or code division multiple access (CDMA) network.The cellular network may include mobile devices, cell sites, basestations, repeaters, antennas, towers, and the like. The cell networkmay be a GSM, GPRS, 3G, EVDO, mesh, or other networks types.

The methods, programs codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic books readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as a flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on a peer topeer network, mesh network, or other communications networks. Theprogram code may be stored on the storage medium associated with theserver and executed by a computing device embedded within the server.The base station may include a computing device and a storage medium.The storage device may store program codes and instructions executed bythe computing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g., USB sticks orkeys), floppy disks, magnetic tape, paper tape, punch cards, standaloneRAM disks, Zip drives, removable mass storage, off-line, and the like;other computer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink, and thelike.

The methods and systems described herein may transform physical and/orintangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices having artificial intelligence, computingdevices, networking equipment, servers, routers, and the like.Furthermore, the elements depicted in the flow chart and block diagramsor any other logical component may be implemented on a machine capableof executing program instructions. Thus, while the foregoing drawingsand descriptions set forth functional aspects of the disclosed systems,no particular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it may beappreciated that the various steps identified and described above may bevaried and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps thereof, may berealized in hardware, software or any combination of hardware andsoftware suitable for a particular application. The hardware may includea general purpose computer and/or dedicated computing device or specificcomputing device or particular aspect or component of a specificcomputing device. The processes may be realized in one or moremicroprocessors, microcontrollers, embedded microcontrollers,programmable digital signal processors or other programmable devices,along with internal and/or external memory. The processes may also, orinstead, be embodied in an application specific integrated circuit, aprogrammable gate array, programmable array logic, or any other deviceor combination of devices that may be configured to process electronicsignals. It may further be appreciated that one or more of the processesmay be realized as a computer executable code capable of being executedon a machine readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions.

Thus, in one aspect, each method described above and combinationsthereof may be embodied in computer executable code that, when executingon one or more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof, and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

While the methods and systems described herein have been disclosed inconnection with certain preferred embodiments shown and described indetail, various modifications and improvements thereon may becomereadily apparent to those skilled in the art. Accordingly, the spiritand scope of the methods and systems described herein are not to belimited by the foregoing examples but is to be understood in thebroadest sense allowable by law.

All documents referenced herein are hereby incorporated by reference intheir entirety.

What is claimed is:
 1. A method of navigating video content, comprising:generating a time-aligned video feed of an event and forming a set offrame-synchronized video segments or frame-synchronized video feeds forthe event; identifying related content of the event from the set offrame-synchronized video segments or video feeds; detecting a swiping ortapping gesture of a user interface of a client device; and navigating adisplay of the set of (related) frame-synchronized video segments orframe-synchronized video feeds in response to the detected swiping ortapping gesture; and wherein in response to a vertical swipe gesture,the navigating is toggled (toggles) between at least two different videocamera angles of a selected video clip.
 2. The method of claim 1,further comprising organizing related content into a set of storylineswith each storyline having a set of storyline elements, with eachstoryline element having an associated screen to display media content.3. The method of claim 2, wherein in response to a detected horizontalswiping gesture, navigating to a next or previous storyline.
 4. Themethod of claim 2, wherein in response to a detected tapping gesture,skipping to a next or previous element within a storyline.
 5. The methodof claim 2, wherein in response to a detected vertical swiping gesture,accessing a different mode for an individual storyline element.
 6. Themethod of claim 5, wherein in response to detecting a vertical swipegesture, the navigation is toggled to display (displays) a discoverscreen over a storyline element screen.
 7. A method of navigating videocontent, comprising: generating a time-aligned video feed of an eventand forming a set of frame-synchronized video segments or video feedsfor the event; identifying related content of the event from the set offrame-synchronized video segments or frame-synchronized video feeds; anddetecting a swiping or tapping gesture of a user interface of a clientdevice; and navigating a display of the set of (related)frame-synchronized video segments or frame-synchronized video feeds inresponse to the detected gesture; wherein in response to detecting aswiping gesture, the navigating switching the display between differentframe-synchronized video streams of the same event from different cameraangles, or different augmented versions of the video streams.
 8. Amethod of navigating video content, comprising: generating atime-aligned video feed of an event and forming a set offrame-synchronized video segments or frame-synchronized video feeds forthe event; identifying related content of the event from the set offrame-synchronized video segments or frame-synchronized video feeds;detecting a swiping or tapping gesture of a user interface of a clientdevice; navigating a display of the set of frame-synchronized videosegments or frame-synchronized video feeds in response to the detectedswiping or tapping gesture; and generating tracking information and aspatial and temporal index for time-aligning the video feeds.
 9. Themethod of claim 7, wherein generating tracking information includestime-aligning the (broadcast) video (feed) feeds with the tracking videofeed based on (the) one or more respective pixel locations and (the) oneor more respective spatial locations.
 10. The method of claim 1, whereinthe gesture is a gesture entered on a touchscreen of a user clientdevice.
 11. A computer implemented method, comprising: forming a set offrame-synchronized video segments or frame-synchronized video feeds forrelated content associated with an event using tracking information anda trained machine learning apparatus to perform spatial and temporalsynchronization; detecting a swiping or tapping gesture of a userinterface of a client device; and navigating a display of the relatedcontent in response to the detected swiping or tapping gesture; andwherein in response to detecting a swiping gesture, the navigatingswitching between different frame-synchronized streams of the same eventfrom at least two different camera angles, or at least two differentaugmented versions of the video streams.
 12. The method of claim 10,wherein related content is organized into storylines with each storylinehaving at least one storyline element to display a video segment; andthe navigating comprises navigating the storyline and storyline elementsin response to the detected swiping or tapping gesture.
 13. The methodof claim 11, wherein the detected swiping or tapping gesture comprises ahorizontal swipe gesture, and navigating the set of related videosegments comprises jumping to a next or previous storyline in responseto detecting the horizontal swipe gesture.
 14. The method of claim 11,wherein the detected swiping or tapping gesture comprises a verticalswipe gesture and navigating the set of related video segmentscomprising toggling between different video clips of a set of storylineelement screens.
 15. The method of claim 11, wherein the detectedswiping or tapping gesture comprises a tap gesture and wherein thenavigating comprises skipping to a next element within a storyline. 16.A computer implemented method, comprising: forming a set offrame-synchronized video segments or frame-synchronized video feeds forrelated content associated with an event using tracking information anda trained machine learning apparatus to perform spatial and temporalsynchronization; detecting a swiping or tapping gesture of a userinterface of a client device; navigating a display of the relatedcontent in response to the detected swiping or tapping gesture;receiving a tracking camera video feed comprising a plurality oftracking video frames being captured by a tracking camera having aposition that is calibrated to a frame of reference; tracking one ormore respective pixel locations of an object detected in one or morerespective broadcast video frames of a broadcast video feed; trackingone or more respective spatial locations of the object based on one ormore respective tracking video frames where the object is detected inthe tracking camera video feed; time-aligning the broadcast video feedwith the tracking video feed based on the one or more respective pixellocations and the one or more respective spatial locations; andgenerating a spatio-temporal index (corresponding to the filmedoccurrence) based on the time-alignment of the broadcast video feed withthe tracking video feed, wherein the spatio-temporal index indexesspatio-temporal information relating to objects detected in thebroadcast video feed and/or the tracking video feed.
 17. The method ofclaim 10, wherein the swiping or tapping gesture is a gesture entered ona touchscreen of a user client device.
 18. A system, comprising: a mediasystem having a tracking system and a trained machine learning systemtrained to receive a video feed and generate a time-aligned video feedof an event, form a set of frame- synchronized video segments orframe-synchronized video feeds for the event, and organize relatedcontent of the event from the set of frame-synchronized video segmentsor frame-synchronized video feeds; the media system including a gesturedetection module to detect a swiping or tapping gesture of a userinterface of a client device and in response navigate a display of theset of (related) frame-synchronized video segments or video feeds inresponse to the detected gesture; and wherein in response to a verticalswipe gesture, the response to navigate the display includes togglingbetween at least two different video camera angles of the same videoclip.
 19. A system, comprising: a media system having a tracking systemand a trained machine learning system trained to receive a video feedand generate a time-aligned video feed of an event, form a set of frame-synchronized video segments or frame-synchronized video feeds for theevent, and organize related content of the event from the set offrame-synchronized video segments or frame-synchronized video feeds; themedia system including a gesture detection module to detect a swiping ortapping gesture of a user interface of a client device and in responsenavigate a display of the set of (related) frame-synchronized videosegments or frame-synchronized video feeds in response to the detectedswiping or tapping gesture; and wherein in response to detecting aswiping gesture, the navigating including switching between differentframe-synchronized streams of the same event from at least two differentcamera angles, or at least two different augmented versions of the videostreams.
 20. A system, comprising: a media system having a trackingsystem and a trained machine learning system trained to receive a videofeed, generate tracking information and a spatial and temporal index fortime-aligning the video feed, and generate a time-aligned video feed ofan event, form a set of frame-synchronized video segments orframe-synchronized video feeds for the event, and organize relatedcontent of the event from the set of frame-synchronized video segmentsor video feeds; and the media system including a gesture detectionmodule to detect a swiping or tapping gesture of a user interface of aclient device and in response navigate a display of the set of (related)frame-synchronized video segments or frame-synchronized video feeds inresponse to the detected gesture.